Decomb Custom syntax

Discuss encoding for devices and presets.
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Decomb Custom syntax

Post by markfilipak »

Description of problem or question:

I'm seeking to fine tune 'Decomb'. I consulted the ffmpeg filters but HB seems to use different nomenclature. I also consulted the HB CLI -h. I also consulted HB documentation (such as it is).

From the GUI, _/Filters\_ Deinterlace='Decomb' Preset='Custom':
Decomb syntax: mode=m:magnitude-thresh=m:variance-thresh=v:laplacian-thresh=l:dilation-thresh=d:erosion-thresh=e:noise-thresh=n:search-distance=s:postproc=p:parity=p

I am truly mystified by that.

Can anyone decipher that for me?
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

I'm seeking to fine tune 'Decomb'.
Custom settings for Decomb are in the Filters UI.
Saving you some time here, compare EEDI2 and Default. Pick one.
There are no better custom settings to be found or contrived.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

Well, actually, I did compare them.

I have a telecined video for which I'd like to 'tone down' the sensitivity of 'Decomb'.

If 'Decomb'=='Off', slow panning shots are good, but when an actor rapidly crosses the frame right in front of the camera, combing stretches half way across the screen.

If 'Decomb'=='Default', the actor crossing is good, but jungle leaves markedly scintillate during slow panning shots.

I seek something in the middle of those two extremes, between 'Off' & 'Default'.

Decomb syntax: mode=m:magnitude-thresh=m:variance-thresh=v:laplacian-thresh=l:dilation-thresh=d:erosion-thresh=e:noise-thresh=n:search-distance=s:postproc=p:parity=p

I've figured out that colon (:) is a separator and that "=m", "=m", "=v", etc. probably mean "=<arg>", but I'm encountering difficulty finding documentation for "mode" & "magnitude-thresh" & "variance-thresh" & "laplacian-thresh" & "dilation-thresh" & "erosion-thresh" & "noise-thresh" & "search-distance" & "postproc" & "parity" -- I have no idea what the <arg>s are. Also, regarding what the parameters do, I can make guesses based on their names, but real documentation would be great. I probably have not looked in the right places. The ffmpeg filter libraries don't seem to use those parameter names.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

Let me show you the problem. Regarding the parameters, here's all that 'handbreak -h' provides:

Code: Select all

   -5, --decomb[=string]   Deinterlace video using a combination of yadif,
                           blend, cubic, or EEDI2 interpolation.
                           Presets:
                               bob
                               eedi2
                               eedi2bob
                           Custom Format:
                               mode=m:magnitude-thresh=m:variance-thresh=v:
                               laplacian-thresh=l:dilation-thresh=d:
                               erosion-thresh=e:noise-thresh=n:
                               search-distance=s:postproc=p:parity=p
                           Default:
                               mode=7
There's no explanations. And "mode=7"? What does that mean?
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

Well, you are in your own zone.
Having tested these algorithms quantitatively and in broad-based commercial production since 2011, we have little understanding of isolated temporal subjectivity, which is a moving target, and as individualized as the source and observer themselves. Are you trying to hit the mythical center of a scatter plot? No one ever won a golf tournament that way.

Here are some current test:
viewtopic.php?f=6&t=38645&p=182238

Maybe this old article will shed some light on your inquiry.
https://www.avsforum.com/forum/26-home- ... rlace.html
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

musicvid wrote: Fri Sep 13, 2019 1:42 am Well, you are in your own zone.
Having tested these algorithms quantitatively and in broad-based commercial production since 2011, we have little understanding of isolated temporal subjectivity, which is a moving target, and as individualized as the source and observer themselves. Are you trying to hit the mythical center of a scatter plot? No one ever won a golf tournament that way.
I love that answer. But, nope. I'm just looking for the meanings of "mode" & "magnitude-thresh" & "variance-thresh" & "laplacian-thresh" & "dilation-thresh" & "erosion-thresh" & "noise-thresh" & "search-distance" & "postproc" & "parity"

And thanks for the links. I'm sure they will help.
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

Your academic curiosity is admirable,
I'm sure other hobbyists will benefit from your insights.
Did you try Decomb=Fast, assuming it is less precise and may give slightly softer edges?
Is your monitor sharpness set maybe a little on the high side?
If you are dealing with interlaced DVD source, as I suspect, it may just be a matter of trying to put mascara on a pig.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

musicvid wrote: Fri Sep 13, 2019 5:19 am Your academic curiosity is admirable,
I'm sure other hobbyists will benefit from your insights.
Did you try Decomb=Fast, assuming it is less precise and may give slightly softer edges?
That's a wonderful idea! Unfortunately, there is no such setting for 'Decomb'. There is an 'Interlace Detection'=='Fast' setting though. I will try that.
Is your monitor sharpness set maybe a little on the high side?
Ummm.... (he says dubiously). I'm previewing simultaneously on a UHD laptop and an HD TV. Neither is displaying any artifacting. I always turn off all automatic settings and calibrate manually. My TV looks like a movie screen: desaturated, reasonable contrast, mile-deep blacks, etc.

If you are dealing with interlaced DVD source, as I suspect, it may just be a matter of trying to put mascara on a pig.[/quote]
The source is hard interlaced. ffprobe says, for example:
  • frames.frame.115.interlaced_frame=1
  • frames.frame.115.top_field_first=1
  • frames.frame.115.repeat_pict=0
however, the origin is movie film (i.e., 24fps). IVTC works just fine. What I think the problem may be is IVTC is being done prior to Decomb. I know that doesn't make sense, but I'm grasping at straws at this point.

By the way. To clear up a previous assertion by John, setting
'Detelecine'=='Default' & 'Interlace Detection'=='Default' (all other filters 'Off')
versus
'Detelecine'=='Default' & 'Interlace Detection'=='Off' (all other filters 'Off')
definitely makes a difference. It's a small difference, but it's discernable. With 'Interlace Detection'=='Off', there's very slight combing.

Not that it makes a big difference but I'm not a hobbiest. I'm an engineer.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

Okay. I have JOY to report.

I was able to find the sweet spot where jungle leaves don't scintillate AND the guy passing quickly in front of the camera doesn't comb. Here's my blow-by-blow accounting.

Bottom line:
'Detelecine'=='Default' & 'Interlace Detection'=='LessSensitive' & 'Deinterlace'=='Decomb' & 'Preset'=='Default' (all other filters 'Off'), i.e.: D.L,DD.O.O,
produced a perfect transcode.

Along the way I tried
'Detelecine'=='Default' & 'Interlace Detection'=='Fast' (all other filters 'Off'), i.e.: D.F,O.O.O, cut down on the combing but not completely, and
'Detelecine'=='Default' & 'Interlace Detection'=='LessSensitive' (all other filters 'Off'), i.e.: D.L,O.O.O, helped more and caused me to conclude that it did actually require some decombing.

The D.L.DD.O.O filter settings are now my go-to settings for hard telecined sources from movie origins.

Thank you musicvid!
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

Recently discovered: The log contains the mysterious parameter list for 'Decomb'. Here's 2 that nearly do what I want.

Does anyone know what the parameters and arguments mean?

Code: Select all

mode=3:spatial-metric=2:motion-thresh=3:spatial-thresh=3:filter-mode=2:block-thresh=40:block-width=16:block-height=16
mode=0:spatial-metric=2:motion-thresh=2:spatial-thresh=3:filter-mode=1:block-thresh=80:block-width=16:block-height=16
The difference between them is the result of one change: 'Interlace Detection'=='LessSensitive' to 'Interlace Detection'=='Fast'. That one change altered 50% of the parameter string.

Now, I'm sure that the parameter strings have been painstakingly worked out by dedicated HB developers and user-experts. It would be nice if the meanings of the arguments were documented so we can all join in on the fun. I would be willing to help attain that goal if there was at least something/anything upon which to base the effort.
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

That's quite a lot of "ado" for mediocre source (480i) but it sounds like you don't have production considerations or deadlines. So, on to the next title! How many do you have?

Is scintillate an engineering term? My point being, I am sure when you have this all settled, you will publish quantified results with test procedures, stable benchmarks, source examples, and PSNR /MSE (at the minimum) to back your impressions, so as to help others with their source, their screens, their eyes, and their viewing preferences.

Guys over at Doom9 might have some interesting responses.
Best.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

musicvid wrote: Fri Sep 13, 2019 2:34 pm That's quite a lot of "ado" for mediocre source (480i) but it sounds like you don't have production considerations or deadlines. So, on to the next title! How many do you have?
About a thousand.
Is scintillate an engineering term? My point being, I am sure when you have this all settled, you will publish quantified results with test procedures, stable benchmarks, source examples, and PSNR /MSE (at the minimum) to back your impressions, so as to help others with their source, their screens, their eyes, and their viewing preferences.
I just have my eyes as test instruments. A couple of years ago I read about PSNR in relation to a discussion I was having in the AVR forum. I thought at the time that PSNR was something that could be measured, so people were measuring it and then despirately trying to find a way to validate it's use. I found their rationales to be questionable at best. Just because something can be measured doesn't mean that the measurements are meaningful. What did impress me were the descriptions of some of the 'blind' screening tests that compared various codecs. I recall that they were conducted somewhere in Russia -- St. Petersburg, I think.

Now that I'm back in the video quality business -- I'm actually retired -- I've been giving some thought to what would make a killer encoder app. Who in the HandBrake universe do you think would be receptive to making some wholesale revisions or a completely new app? It would need to have a built-in player. If you're interested in hearing what I've been thinking, I'd be happy to share, but I don't want to assume interest and barrage you with unwelcome blabber.
Guys over at Doom9 might have some interesting responses.
I'm familiar with Doom9. Do you have any particular sub-forum in mind?

On topic: Was it you who said that MPV with deinterlace 'on' (the 'd' key) uses yadif?
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

Your eyes are a stable instrument, and bitwise quantification is not?
What did impress me were the descriptions of some of the 'blind' screening tests that compared various codecs. I recall that they were conducted somewhere in Russia -- St. Petersburg, I think.
You are referring to the highly regarded benchmarking that has been going on at MSU since interframe compression was a "thing," nearly twenty years ago. I suggest that is the place for you to start your learning curve. Their perceptually-weighted SSIM technology is about the best there is. https://www.compression.ru/video/

I fail to understand your apparent mystification of PSNR. It is an inverse log function of ordinary RMS Regression theory, which I taught in tenth grade in this state. Cleverly expressed on a 0-100 dB log scale, it gives an immediate and clear linear visual reference to output noise caused by filters, among many other things. Add to that MSU's pioneering work with SSIM, which is perceptually weighted, and you may consider giving your wayward eyes a rest.

Back to topic one final time, Decomb=Default is your friend. QTGMC and EEDI2 are both slower, and have sharper edges, which you have said you do not prefer. If I found a gold nugget laying on the ground, I would pick it up; and not start digging out of spite.

That's really all I can add to the discussion; eagerly awaiting the outcomes of your quantified testing.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

musicvid wrote: Fri Sep 13, 2019 3:58 pm Your eyes are a stable instrument, and bitwise quantification is not?
What did impress me were the descriptions of some of the 'blind' screening tests that compared various codecs. I recall that they were conducted somewhere in Russia -- St. Petersburg, I think.
You are referring to the highly regarded benchmarking that has been going on at MSU since interframe compression was a "thing," nearly twenty years ago. I suggest that is the place for you to start your learning curve. Their perceptually-weighted SSIM technology is about the best there is. https://www.compression.ru/video/
Yes. That's it.
I fail to understand your apparent mystification of PSNR. It is an inverse log function of ordinary RMS Regression theory, which I taught in tenth grade in this state. Cleverly expressed on a 0-100 dB log scale, it gives an immediate and clear linear visual reference to output noise caused by filters, among many other things. Add to that MSU's pioneering work with SSIM, which is perceptually weighted, and you may consider giving your wayward eyes a rest.
I'm not mystified by SNR. I just think that it's a very small part of video quality.
Back to topic one final time, Decomb=Default is your friend. QTGMC and EEDI2 are both slower, and have sharper edges, which you have said you do not prefer. If I found a gold nugget laying on the ground, I would pick it up; and not start digging out of spite.
Well, in the same group of 480p30 special features for a particular BD movie, I've run across another case where MPV's deinterlace is superior to HB's deinterlace. In this case, however, I'm getting pretty severe judder with HB. I'm trying all variations of decomb and yadif. This is getting tedious.

I mean, this is just physics. I don't understand why I'm getting such varying results with what is essentially the same videos: hard telecine from movie source. HB has to be diddling in the background, trying to 'help' me.
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

[Duplicate content deleted]
Last edited by Deleted User 13735 on Fri Sep 13, 2019 4:59 pm, edited 1 time in total.
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

I think you should stick with MPV and avoid making comparisons. The developers here do become mildly annoyed with repeated references to derived software, just a heads up.
Well, in the same group of 480p30 special features for a particular BD movie, I've run across another case where MPV's deinterlace is superior to HB's deinterlace
You are deinterlacing progressive source? I thought that was explained to you. I'm seeing some red flags, so I'll just encourage you to keep studying and above all, avoid the common tendency of misattribution. Best.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

musicvid wrote: Fri Sep 13, 2019 4:46 pm
Well, in the same group of 480p30 special features for a particular BD movie, I've run across another case where MPV's deinterlace is superior to HB's deinterlace
You are deinterlacing progressive source?
No, the source is hard telecined i30. (I'm not surprised you're getting confused.) When I watch the i30 source, the player's deinterlace produces no judder. When I watch the p24 target, it judders -- turning on deinterlace in the player of course does nothing to p24. I've made about a dozen runs, 4 with manual custom yadif settings. I'm giving up and taking the one with the least judder. I have not found any patterns of goodness that helps to select better manual yadiff settings. BTW, the source is 480i30 hard, but it's got good bandwidth because it's on a BD. The picture is stunning. It's a Werner Herzog movie, so you know it's very cinematic.
Deleted User 13735

Re: Decomb Custom syntax

Post by Deleted User 13735 »

Yes, I'm quite certain that one of us is confused.
But my participation is finished; you see, I have a production deadline to meet, converting some old kinescopes for archiving
;?)
mduell
Veteran User
Posts: 8187
Joined: Sat Apr 21, 2007 8:54 pm

Re: Decomb Custom syntax

Post by mduell »

So many words spilled on the page, so little accomplished.

Regarding the options you're asking about, have you perused the decomb sourcecode yet? It should be readily apparent they're EEDI options, which will easily lead you to the EEDI documentation via Google.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

mduell wrote: Fri Sep 13, 2019 10:29 pm So many words spilled on the page, so little accomplished.
You just spilled more words, my friend. And you're severely mistaken about the "little accomplished" part. Just who are you to judge, eh?
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: Decomb Custom syntax

Post by BradleyS »

Some information on the mode enum is available as comments in the source, near the top: https://github.com/HandBrake/HandBrake/ ... b/decomb.c

The thresholds are how many samples and such must be identified to be combed in order for the frame to be treated as such. The block sizes determine how the frame is split up into chunks for analysis.

The current default errs slightly toward false positives for combing which ensures all combing is caught and processed. Less Sensitive (the old default) errs slightly toward false negatives and occasionally allows very small amounts of combing through. The current default is the best we've found in testing, yet both can sometimes match on very fine combing-like textures such as copious amounts of hair (monsters) and foliage.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

Good of you to help.
BradleyS wrote: Sun Sep 15, 2019 11:26 pm Some information on the mode enum is available as comments in the source, near the top: https://github.com/HandBrake/HandBrake/ ... b/decomb.c
Thanks. I've coded in about 14 languages, but other people have always written 'C' for me. I guess I've assumed notes on static vars are always with structs and are always in header files. My bad.
The thresholds are how many samples and such must be identified to be combed in order for the frame to be treated as such. The block sizes determine how the frame is split up into chunks for analysis.
Well, I guess I've been naively thinking that countering combing is strictly a mechanical process, strictly a matter of physics, that if I knew enough about the video source, analysis & tweeking was unneccessary, that if that wasn't enough to defeat combing, it was because the original author screwed the pooch.
The current default errs slightly toward false positives...
That's golden info. Thanks!

You boys & girls don't IRC, eh? I've been to #handbrake on freenode many times and I'm always the only one there. Or is it that I don't know how to run HexChat? Seems to work for other channels. Please advise.
mduell
Veteran User
Posts: 8187
Joined: Sat Apr 21, 2007 8:54 pm

Re: Decomb Custom syntax

Post by mduell »

markfilipak wrote: Mon Sep 16, 2019 1:55 am
The thresholds are how many samples and such must be identified to be combed in order for the frame to be treated as such. The block sizes determine how the frame is split up into chunks for analysis.
Well, I guess I've been naively thinking that countering combing is strictly a mechanical process, strictly a matter of physics, that if I knew enough about the video source, analysis & tweeking was unneccessary, that if that wasn't enough to defeat combing, it was because the original author screwed the pooch.
You only have combing if you have motion; without motion you can treat the interlaced fields as a progressive frame. The analysis and tweaking is balanced against how much motion is worth the quality loss of deinterlacing.
markfilipak
Bright Spark User
Posts: 313
Joined: Thu Aug 01, 2019 8:58 pm

Re: Decomb Custom syntax

Post by markfilipak »

mduell wrote: Mon Sep 16, 2019 3:51 pmYou only have combing if you have motion; without motion you can treat the interlaced fields as a progressive frame.
I think just about everybody knows that.
The analysis and tweaking is balanced against how much motion is worth the quality loss of deinterlacing.
"quality loss of deinterlacing"? Please explain.
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: Decomb Custom syntax

Post by BradleyS »

The beauty of interlacing detection is where no combing artifacts are detected, the frame passes untouched, preserving resolution and sharpness that is typically diminished by any deinterlacing algorithm.
Post Reply