Page 2 of 3

Re: NLMeans denoise filter

Posted: Fri Oct 31, 2014 2:51 pm
by JohnAStebbins
Kytael wrote:have any steps taken towards the OpenCL implementation been taken? I can't find any other than a small mention on the locked 0.10.0 beta thread: viewtopic.php?f=6&t=30613&hilit=nlmeans#p142091
There is an OpenCL version that seems to work quite well, more info and source code can be found here: NLMeansCL: GPU based Non Local Means Denoising
while following this link there is also a paper describing a GPU-accelerated real-time NLMeans algorithm: A GPU-accelerated real-time NLMeans algorithm .
No

Re: NLMeans denoise filter

Posted: Sat Nov 01, 2014 12:04 am
by BradleyS
It's worth mentioning that there are hundreds of variations on NLMeans filtering, perhaps because the core concept is solid but leaves a lot of room for improvement. HandBrake's implementation is not absolutely the same as any other, including any OpenCL implementation. Additionally, the implementation HandBrake's NLMeans is based on is properly GPL licensed with the consent of the patent holders.

Attempting to copy and paste another implementation will likely not provide the same API and may not clearly meet the same legal standard. Contributions are welcome but for these reasons, any OpenCL efforts made starting from the C source will likely be more viable than ones starting from available OpenCL libraries, as counter-intuitive as this may initially seem.

Re: NLMeans denoise filter

Posted: Sun Nov 02, 2014 12:59 am
by Djfe
So for the legal aspect: If I would start working on an open-cl version of it and would only base it on the GPL licensed NLMeans code, then I wouldn't need to ask the patent holders/wouldn't have any legal issues?

Re: NLMeans denoise filter

Posted: Sun Nov 02, 2014 1:19 am
by BradleyS
I am not a lawyer and so far I've been unsuccessful at predicting the future. Case in point, I did not predict that my previous comments would open a legal can of worms, so to speak.

Anyway, I only meant to point out that I developed HandBrake's implementation of NLMeans in good faith that legal requirements have been satisfied—and being GPL, I imagine any derivative implementation made in good faith would be on equal footing. I am not aware of the legal status of other implementations, only that some have been created under different licenses and as far as I know, make no mention of the related patents.

The more immediate issue is, our OpenCL code base is not maintained. Who will maintain it? If you or someone else rewrites filters in OpenCL, who will maintain those? I don't mean to discourage you or anyone else from contributing; I only wish to point out that meaningfully contributing requires more than simply writing some code to do a particular thing (or do it faster).

Re: NLMeans denoise filter

Posted: Mon Nov 03, 2014 5:16 pm
by Djfe
It was just an idea, If I would start learning OpenCL in the future I would have a project I could contribute to :)
No promises here... (just an idea)

Re: NLMeans denoise filter

Posted: Mon Nov 10, 2014 6:16 pm
by Kytael
BradleyS wrote:It's worth mentioning that there are hundreds of variations on NLMeans filtering, perhaps because the core concept is solid but leaves a lot of room for improvement. HandBrake's implementation is not absolutely the same as any other, including any OpenCL implementation. Additionally, the implementation HandBrake's NLMeans is based on is properly GPL licensed with the consent of the patent holders.

Attempting to copy and paste another implementation will likely not provide the same API and may not clearly meet the same legal standard. Contributions are welcome but for these reasons, any OpenCL efforts made starting from the C source will likely be more viable than ones starting from available OpenCL libraries, as counter-intuitive as this may initially seem.
sure, I wasn't suggesting the code be copy-pasted, just wondering about progress. I imagine any code gotten from anywhere else would have to be heavily modified for use in handbrake. nevertheless, the context is helpful.

as far as licenses, the avisynth version appears to be GPLv2-licensed.
darktable, an open-source photo processor, has a GPLv3 licensed C implementation. I've seen GPLv3 code alongside GPLv2 code before, but it seems questionable to me, what's the policy for handbrake?

additionally, is an implementation more or less from scratch preferred? what did you base the current implementation on?
BradleyS wrote: The more immediate issue is, our OpenCL code base is not maintained. Who will maintain it? If you or someone else rewrites filters in OpenCL, who will maintain those? I don't mean to discourage you or anyone else from contributing; I only wish to point out that meaningfully contributing requires more than simply writing some code to do a particular thing (or do it faster).
good point. did the last OpenCL dev disappear?

Re: NLMeans denoise filter

Posted: Mon Nov 10, 2014 6:29 pm
by Rodeo
There were 3 developers in total, and all of them are out of reach for various reasons.

Re: NLMeans denoise filter

Posted: Tue Nov 11, 2014 3:05 am
by Djfe
The current NLMeans implementation is based on the FFMPEG code that was committed by Dirk Farin
the link to his website was in the first post of this thread:
http://dirk-farin.net/projects/nlmeans/index.html

at the end there are two references

as far as I know he has asked the patent holders for permission before publishing his implementation of it.


Whether the devs of the code you have refered to have done something similar so that they were allowed to use that knowledge is unknown to us, that's why the current implementation is preferred

If I have said something wrong, then pls correct me ;)

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 1:52 pm
by nhyone
I was going to ask what the strength presets (ultralight, light, medium and strong) means, but luckily I decided to read the source code first.

It wasn't clear to me, but this affects the strength only. Duh.

Ultralight = 1.5
Light = 3
Strong = 10
Default = 6 (presumably medium)

The doc says default is 8, though.

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 2:16 pm
by BradleyS
The CLI default strength with no parameters or tune specified is 8, for a full default string of 8:1:7:3:2:0 as specified in this outdated thread. A recent code addition changed this (only applies to development version post-0.10.x); I'll see if I can get some clarity on the matter.

It's not entirely reasonable to simply list the strengths this way. Strength varies by both preset *and* tune, and the realized strength depends on all of the other parameters. Additionally, most of the presets use different strengths for luma and chroma.

Anyway, I would list the entire set of preset/tune combinations, but there are twenty of them and their real purpose is "set it and forget it". The curious can indeed inspect generate_nlmeans_settings() in libhb/param.c. Definitely understand what each parameter does, since they all affect each other quite a bit.

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 3:02 pm
by nhyone
You're right. I only looked at tune none. The others affect origin weight or frames (for animation).

I won't call this thread out-of-date. It is a sticky and the only place that explains what the parameters mean. :D

But of course, if the author says it's outdated...

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 3:08 pm
by BradleyS
Well the last post prior to today was in a previous year... :D

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 8:33 pm
by BradleyS
Some clarity (first post updated):

0.10.x
The CLI default for --nlmeans with no parameters specified is 8:1:7:3:2:0.

Post 0.10.x (revision 7402 onward)
The CLI default for --nlmeans with no parameters specified is now the same as --nlmeans="medium" (with or without --nlmeans-tune="none"), or 6:1:7:3:2:0. When adding preset import/export to the CLI we accidentally made light the default; this is now fixed and standardized on medium.

Re: NLMeans denoise filter

Posted: Mon Aug 17, 2015 8:58 pm
by JohnAStebbins
Bradley is being kind when he says it was "accidental" :wink: I made a change and never circled back around with him to verify that it was ok :mrgreen:

Re: NLMeans denoise filter

Posted: Tue Aug 18, 2015 12:54 pm
by nhyone
I think it is useful to know how the tune presets map to the underlying numeric values.

Code: Select all

            u/light                  light            medium        strong
none        1.5    :1  :7:3  :2  :0  3     :          6  :          10    :
film        1.5/2.4:0.9:7:3  :2  :0  3/4   :0.9    :  6/8:0.8    :  8/10  :0.6    :
grain       0/2.4  :0.9:7:3  :2  :0  0/3.5 :0.9    :  0/6:0.8    :  0/8   :0.6    :
highmotion  1.5/2.4:0.9:7:3/5:2/1:0  3/3.25:0.9/0.8:  6  :0.8/0.7:  8/6.75:0.6/0.5:

            u/light                  light            medium        strong
none        1.5  :1   :7:3:2:0       3     :          6  :          10  :
animation   2.5/2:0.15:5:7:2:0       3/2.25::::3:     5/4::::4:     10/8::::4:
All six parameters are shown for ultralight. For light, medium and strong, only the different values are shown. For all, only the first two parameters (strength and origin weight) are different, except for animation, which varies on strength and frames (the fifth parameter).

For parameters with '/', the first value is luma and the second chroma. (I think.)

animation is shown separately cos it makes the lines too long, causing them to wrap and break the formatting.

Re: NLMeans denoise filter

Posted: Tue Aug 18, 2015 5:36 pm
by BradleyS
If you insist, I'll at least provide a less confusing table. Film, grain, and high motion are all related so I put them on their own axis to avoid running off the page.

As of release version 0.10.x and development version 7402 (2015-08-17):

Code: Select all

            none           film                 animation
ultralight  1.5:1:7:3:2:0  1.5:0.9:7:3:2:0:2.4  2.5:0.15:5:7:2:0:2.00
light       3.0:1:7:3:2:0  3.0:0.8:7:3:2:0:4.0  3.0:0.15:5:7:3:0:2.25
medium      6.0:1:7:3:2:0  6.0:0.8:7:3:2:0:8.0  5.0:0.15:5:7:4:0:4.00
strong       10:1:7:3:2:0  8.0:0.6:7:3:2:0:10    10:0.15:5:7:4:0:8.00

                           grain
ultralight                 0.0:0.9:7:3:2:0:2.4 (same as 0:0:0:0:0:0:2.4:0.9:7:3:2:0)
light                      0.0:0.8:7:3:2:0:3.5 (same as 0:0:0:0:0:0:3.5:0.8:7:3:2:0)
medium                     0.0:0.8:7:3:2:0:6.0 (same as 0:0:0:0:0:0:6.0:0.8:7:3:2:0)
strong                     0.0:0.6:7:3:2:0:8.0 (same as 0:0:0:0:0:0:8.0:0.6:7:3:2:0)

                           high motion
ultralight                 1.5:0.9:7:3:2:0:2.40:0.9:7:5:1:0
light                      3.0:0.9:7:3:2:0:3.25:0.8:7:5:1:0
medium                     6.0:0.8:7:3:2:0:6.00:0.7:7:5:1:0
strong                     8.0:0.6:7:3:2:0:6.75:0.5:7:5:1:0

Re: NLMeans denoise filter

Posted: Tue Aug 18, 2015 6:06 pm
by Rodeo
Might be worth updating the first post?

Re: NLMeans denoise filter

Posted: Tue Aug 18, 2015 6:12 pm
by BradleyS
Rodeo wrote:Might be worth updating the first post?
Done.

Re: NLMeans denoise filter

Posted: Tue May 03, 2016 3:43 am
by BradleyS
New NLMeans tunes will be in the next nightly build. Tape is for analog tape sources including VHS, and Sprite is for most 1-/4-/8-/16-bit 2D games.

Tape is about 50% faster than Film and in addition to analog tape sources, may be useful on lower quality standard definition sources such as DVD transfers of old films. Old black and white and Technicolor-era sources look great. I've even used it successfully on old cartoons that were shot on film and later transferred to DVD. Animation is more thorough in that scenario, but Tape looks more natural to me for some film-era animation. For modern animation, definitely stick to the Animation tune.

Sprite is useful for cleaning up the hard edges, simple colors, and repeating patterns typically found in 2D graphics. Your NES recordings will thank you. Don't use Sprite on high definition sources unless you have a lot of time on your hands.

Re: NLMeans denoise filter

Posted: Thu Jul 28, 2016 12:47 pm
by Djfe
BradleyS wrote:Future improvement

While quite advanced, the algorithm has minor quirks that provide some room for improvement.
  • Replacing the current strength metric with one that takes patch size, search range, and temporal depth into account would create more consistency between settings and improve usability.
  • Optimizing to not recalculate patch difference weight(b,a) where weight(a,b) has already been calculated could provide up to 2x speed improvement.
  • Maintaining a covariance matrix would provide a method for disposing of calculations deviating far enough from the mean to lower quality, solving the mild problem of cumulative error associated with the algorithm's weight system. Prefiltering partially solves this problem for noisier sources.
  • Replacing the semi-exhaustive search window with a more intelligent, predictive search would potentially increase performance and quality.
Hi,

What of the tasks in above's list are still on the todo list?
The 2nd task seems promising.

And it would be awesome if you could add tape and sprite settings to the code section in the 1st thread :)

Regards,
Djfe

Re: NLMeans denoise filter

Posted: Thu Jul 28, 2016 1:21 pm
by BradleyS
Realistically, none of the above are planned for the current NLMeans. Should I attempt a new version (NLMeans 2 or the like), I would like to implement the last idea.

As it turns out, NLMeans is limited by the speed of the memory bus and the amount of memory necessary for computation. So the second idea won't provide much benefit without other improvements, (I'm not sure what, yet). I believe the third idea will likely slow things down.

The new documentation project will eventually supersede the information in this thread, and NLMeans details will be added to the technical section. See https://handbrake.fr/docs

Cheers

Re: NLMeans denoise filter

Posted: Thu Jul 28, 2016 1:56 pm
by Djfe
ok, cool
the new docs section looks promising
thx for your quick answer :)

one last question:
Does Handbrake denoise first or deinterlace? I read on the internet, that deinterlacers work better on denoised content
And can I output interlaced in anyway or do I have to use ffmpeg for that then?

I would like to use NL-Means and then QTGMC and then encode with Handbrake
The version of NL-Means in ffmpeg is still the same as in Handbrake, so I could use ffmpeg for NL-means, right?

I'm trying to digitalize my tape collection

Re: NLMeans denoise filter

Posted: Thu Jul 28, 2016 2:07 pm
by BradleyS
Conventional wisdom is that one must denoise before everything else. In practice, our combing detection works fine on noisy sources, and denoising with an averaging filter like NLMeans could actually make things worse. It feels wrong to me to say we don't denoise first, but we don't and it works great. So conventional wisdom is dated.

Detelecine > Comb detect > Decomb/Deinterlace > Denoise

HandBrake's version of NLMeans is not identical to any other implementation. It is based on Dr. Farin's helpful ffmpeg implementation. I've made changes, additions, and what I believe to be some improvements.

HandBrake's output is progressive. All displays going forward are and will be progressive, and software deinterlacers like ours are better than most displays/TVs. So you may want to Decomb/Deinterlace with HandBrake rather than attempt to produce interlaced video (which *will* be deinterlaced later on playback!). Decomb with default settings is usually fine, EEDI2 is sometimes higher quality but slow, and bob is good if you want to preserve smooth motion for that 6 o'clock news look. :)

Re: NLMeans denoise filter

Posted: Thu Aug 25, 2016 12:11 am
by Registered55
EEDI2 Is extremely slow.... but one day in hope we may get something like a yadifmod - NNEDI 3 combo "Fast, efficient and great quality for the speed that it runs at"

Also it should be noted that there doesn't seem to be any (good) anti-bobbing filter used in handbrake to fix the awful issue that comes with all "BOB-bing" techniques.

It's just my view of course, but the bobbing effect more than outweighs any benefits that handbrakes bob deinterlacer gives.... but in fairness the BOB deinterlacing technique was created a very long time ago....LOL. It would be great however if there was some kind of anti-bobbing in handbrake to help reduce the effect of the BOB deinterlacer.

note... don't use EEDI2 with NLMeans unless you have a really really strong CPU...ouch.

Re: NLMeans denoise filter

Posted: Thu Aug 25, 2016 6:12 am
by Djfe
one thing I've wondered for a long time:
why would you want to use decombing instead of deinterlacing if you know the source is interlaced?