NLMeans denoise filter

Archive of historical development discussions
Discussions / Development has moved to GitHub
Forum rules
*******************************
Please be aware we are now using GitHub for issue tracking and feature requests.
- This section of the forum is now closed to new topics.

*******************************
User avatar
JohnAStebbins
HandBrake Team
Posts: 5709
Joined: Sat Feb 09, 2008 7:21 pm

Re: NLMeans denoise filter

Post by JohnAStebbins »

Kytael wrote:have any steps taken towards the OpenCL implementation been taken? I can't find any other than a small mention on the locked 0.10.0 beta thread: viewtopic.php?f=6&t=30613&hilit=nlmeans#p142091
There is an OpenCL version that seems to work quite well, more info and source code can be found here: NLMeansCL: GPU based Non Local Means Denoising
while following this link there is also a paper describing a GPU-accelerated real-time NLMeans algorithm: A GPU-accelerated real-time NLMeans algorithm .
No
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

It's worth mentioning that there are hundreds of variations on NLMeans filtering, perhaps because the core concept is solid but leaves a lot of room for improvement. HandBrake's implementation is not absolutely the same as any other, including any OpenCL implementation. Additionally, the implementation HandBrake's NLMeans is based on is properly GPL licensed with the consent of the patent holders.

Attempting to copy and paste another implementation will likely not provide the same API and may not clearly meet the same legal standard. Contributions are welcome but for these reasons, any OpenCL efforts made starting from the C source will likely be more viable than ones starting from available OpenCL libraries, as counter-intuitive as this may initially seem.
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

So for the legal aspect: If I would start working on an open-cl version of it and would only base it on the GPL licensed NLMeans code, then I wouldn't need to ask the patent holders/wouldn't have any legal issues?
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

I am not a lawyer and so far I've been unsuccessful at predicting the future. Case in point, I did not predict that my previous comments would open a legal can of worms, so to speak.

Anyway, I only meant to point out that I developed HandBrake's implementation of NLMeans in good faith that legal requirements have been satisfied—and being GPL, I imagine any derivative implementation made in good faith would be on equal footing. I am not aware of the legal status of other implementations, only that some have been created under different licenses and as far as I know, make no mention of the related patents.

The more immediate issue is, our OpenCL code base is not maintained. Who will maintain it? If you or someone else rewrites filters in OpenCL, who will maintain those? I don't mean to discourage you or anyone else from contributing; I only wish to point out that meaningfully contributing requires more than simply writing some code to do a particular thing (or do it faster).
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

It was just an idea, If I would start learning OpenCL in the future I would have a project I could contribute to :)
No promises here... (just an idea)
Kytael
Posts: 2
Joined: Thu Oct 30, 2014 9:30 pm

Re: NLMeans denoise filter

Post by Kytael »

BradleyS wrote:It's worth mentioning that there are hundreds of variations on NLMeans filtering, perhaps because the core concept is solid but leaves a lot of room for improvement. HandBrake's implementation is not absolutely the same as any other, including any OpenCL implementation. Additionally, the implementation HandBrake's NLMeans is based on is properly GPL licensed with the consent of the patent holders.

Attempting to copy and paste another implementation will likely not provide the same API and may not clearly meet the same legal standard. Contributions are welcome but for these reasons, any OpenCL efforts made starting from the C source will likely be more viable than ones starting from available OpenCL libraries, as counter-intuitive as this may initially seem.
sure, I wasn't suggesting the code be copy-pasted, just wondering about progress. I imagine any code gotten from anywhere else would have to be heavily modified for use in handbrake. nevertheless, the context is helpful.

as far as licenses, the avisynth version appears to be GPLv2-licensed.
darktable, an open-source photo processor, has a GPLv3 licensed C implementation. I've seen GPLv3 code alongside GPLv2 code before, but it seems questionable to me, what's the policy for handbrake?

additionally, is an implementation more or less from scratch preferred? what did you base the current implementation on?
BradleyS wrote: The more immediate issue is, our OpenCL code base is not maintained. Who will maintain it? If you or someone else rewrites filters in OpenCL, who will maintain those? I don't mean to discourage you or anyone else from contributing; I only wish to point out that meaningfully contributing requires more than simply writing some code to do a particular thing (or do it faster).
good point. did the last OpenCL dev disappear?
Deleted User 11865

Re: NLMeans denoise filter

Post by Deleted User 11865 »

There were 3 developers in total, and all of them are out of reach for various reasons.
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

The current NLMeans implementation is based on the FFMPEG code that was committed by Dirk Farin
the link to his website was in the first post of this thread:
http://dirk-farin.net/projects/nlmeans/index.html

at the end there are two references

as far as I know he has asked the patent holders for permission before publishing his implementation of it.


Whether the devs of the code you have refered to have done something similar so that they were allowed to use that knowledge is unknown to us, that's why the current implementation is preferred

If I have said something wrong, then pls correct me ;)
nhyone
Bright Spark User
Posts: 252
Joined: Fri Jul 24, 2015 4:13 am

Re: NLMeans denoise filter

Post by nhyone »

I was going to ask what the strength presets (ultralight, light, medium and strong) means, but luckily I decided to read the source code first.

It wasn't clear to me, but this affects the strength only. Duh.

Ultralight = 1.5
Light = 3
Strong = 10
Default = 6 (presumably medium)

The doc says default is 8, though.
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

The CLI default strength with no parameters or tune specified is 8, for a full default string of 8:1:7:3:2:0 as specified in this outdated thread. A recent code addition changed this (only applies to development version post-0.10.x); I'll see if I can get some clarity on the matter.

It's not entirely reasonable to simply list the strengths this way. Strength varies by both preset *and* tune, and the realized strength depends on all of the other parameters. Additionally, most of the presets use different strengths for luma and chroma.

Anyway, I would list the entire set of preset/tune combinations, but there are twenty of them and their real purpose is "set it and forget it". The curious can indeed inspect generate_nlmeans_settings() in libhb/param.c. Definitely understand what each parameter does, since they all affect each other quite a bit.
nhyone
Bright Spark User
Posts: 252
Joined: Fri Jul 24, 2015 4:13 am

Re: NLMeans denoise filter

Post by nhyone »

You're right. I only looked at tune none. The others affect origin weight or frames (for animation).

I won't call this thread out-of-date. It is a sticky and the only place that explains what the parameters mean. :D

But of course, if the author says it's outdated...
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

Well the last post prior to today was in a previous year... :D
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

Some clarity (first post updated):

0.10.x
The CLI default for --nlmeans with no parameters specified is 8:1:7:3:2:0.

Post 0.10.x (revision 7402 onward)
The CLI default for --nlmeans with no parameters specified is now the same as --nlmeans="medium" (with or without --nlmeans-tune="none"), or 6:1:7:3:2:0. When adding preset import/export to the CLI we accidentally made light the default; this is now fixed and standardized on medium.
User avatar
JohnAStebbins
HandBrake Team
Posts: 5709
Joined: Sat Feb 09, 2008 7:21 pm

Re: NLMeans denoise filter

Post by JohnAStebbins »

Bradley is being kind when he says it was "accidental" :wink: I made a change and never circled back around with him to verify that it was ok :mrgreen:
nhyone
Bright Spark User
Posts: 252
Joined: Fri Jul 24, 2015 4:13 am

Re: NLMeans denoise filter

Post by nhyone »

I think it is useful to know how the tune presets map to the underlying numeric values.

Code: Select all

            u/light                  light            medium        strong
none        1.5    :1  :7:3  :2  :0  3     :          6  :          10    :
film        1.5/2.4:0.9:7:3  :2  :0  3/4   :0.9    :  6/8:0.8    :  8/10  :0.6    :
grain       0/2.4  :0.9:7:3  :2  :0  0/3.5 :0.9    :  0/6:0.8    :  0/8   :0.6    :
highmotion  1.5/2.4:0.9:7:3/5:2/1:0  3/3.25:0.9/0.8:  6  :0.8/0.7:  8/6.75:0.6/0.5:

            u/light                  light            medium        strong
none        1.5  :1   :7:3:2:0       3     :          6  :          10  :
animation   2.5/2:0.15:5:7:2:0       3/2.25::::3:     5/4::::4:     10/8::::4:
All six parameters are shown for ultralight. For light, medium and strong, only the different values are shown. For all, only the first two parameters (strength and origin weight) are different, except for animation, which varies on strength and frames (the fifth parameter).

For parameters with '/', the first value is luma and the second chroma. (I think.)

animation is shown separately cos it makes the lines too long, causing them to wrap and break the formatting.
Last edited by nhyone on Wed Aug 19, 2015 12:35 am, edited 1 time in total.
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

If you insist, I'll at least provide a less confusing table. Film, grain, and high motion are all related so I put them on their own axis to avoid running off the page.

As of release version 0.10.x and development version 7402 (2015-08-17):

Code: Select all

            none           film                 animation
ultralight  1.5:1:7:3:2:0  1.5:0.9:7:3:2:0:2.4  2.5:0.15:5:7:2:0:2.00
light       3.0:1:7:3:2:0  3.0:0.8:7:3:2:0:4.0  3.0:0.15:5:7:3:0:2.25
medium      6.0:1:7:3:2:0  6.0:0.8:7:3:2:0:8.0  5.0:0.15:5:7:4:0:4.00
strong       10:1:7:3:2:0  8.0:0.6:7:3:2:0:10    10:0.15:5:7:4:0:8.00

                           grain
ultralight                 0.0:0.9:7:3:2:0:2.4 (same as 0:0:0:0:0:0:2.4:0.9:7:3:2:0)
light                      0.0:0.8:7:3:2:0:3.5 (same as 0:0:0:0:0:0:3.5:0.8:7:3:2:0)
medium                     0.0:0.8:7:3:2:0:6.0 (same as 0:0:0:0:0:0:6.0:0.8:7:3:2:0)
strong                     0.0:0.6:7:3:2:0:8.0 (same as 0:0:0:0:0:0:8.0:0.6:7:3:2:0)

                           high motion
ultralight                 1.5:0.9:7:3:2:0:2.40:0.9:7:5:1:0
light                      3.0:0.9:7:3:2:0:3.25:0.8:7:5:1:0
medium                     6.0:0.8:7:3:2:0:6.00:0.7:7:5:1:0
strong                     8.0:0.6:7:3:2:0:6.75:0.5:7:5:1:0
Deleted User 11865

Re: NLMeans denoise filter

Post by Deleted User 11865 »

Might be worth updating the first post?
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

Rodeo wrote:Might be worth updating the first post?
Done.
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

New NLMeans tunes will be in the next nightly build. Tape is for analog tape sources including VHS, and Sprite is for most 1-/4-/8-/16-bit 2D games.

Tape is about 50% faster than Film and in addition to analog tape sources, may be useful on lower quality standard definition sources such as DVD transfers of old films. Old black and white and Technicolor-era sources look great. I've even used it successfully on old cartoons that were shot on film and later transferred to DVD. Animation is more thorough in that scenario, but Tape looks more natural to me for some film-era animation. For modern animation, definitely stick to the Animation tune.

Sprite is useful for cleaning up the hard edges, simple colors, and repeating patterns typically found in 2D graphics. Your NES recordings will thank you. Don't use Sprite on high definition sources unless you have a lot of time on your hands.
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

BradleyS wrote:Future improvement

While quite advanced, the algorithm has minor quirks that provide some room for improvement.
  • Replacing the current strength metric with one that takes patch size, search range, and temporal depth into account would create more consistency between settings and improve usability.
  • Optimizing to not recalculate patch difference weight(b,a) where weight(a,b) has already been calculated could provide up to 2x speed improvement.
  • Maintaining a covariance matrix would provide a method for disposing of calculations deviating far enough from the mean to lower quality, solving the mild problem of cumulative error associated with the algorithm's weight system. Prefiltering partially solves this problem for noisier sources.
  • Replacing the semi-exhaustive search window with a more intelligent, predictive search would potentially increase performance and quality.
Hi,

What of the tasks in above's list are still on the todo list?
The 2nd task seems promising.

And it would be awesome if you could add tape and sprite settings to the code section in the 1st thread :)

Regards,
Djfe
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

Realistically, none of the above are planned for the current NLMeans. Should I attempt a new version (NLMeans 2 or the like), I would like to implement the last idea.

As it turns out, NLMeans is limited by the speed of the memory bus and the amount of memory necessary for computation. So the second idea won't provide much benefit without other improvements, (I'm not sure what, yet). I believe the third idea will likely slow things down.

The new documentation project will eventually supersede the information in this thread, and NLMeans details will be added to the technical section. See https://handbrake.fr/docs

Cheers
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

ok, cool
the new docs section looks promising
thx for your quick answer :)

one last question:
Does Handbrake denoise first or deinterlace? I read on the internet, that deinterlacers work better on denoised content
And can I output interlaced in anyway or do I have to use ffmpeg for that then?

I would like to use NL-Means and then QTGMC and then encode with Handbrake
The version of NL-Means in ffmpeg is still the same as in Handbrake, so I could use ffmpeg for NL-means, right?

I'm trying to digitalize my tape collection
User avatar
BradleyS
Moderator
Posts: 1860
Joined: Thu Aug 09, 2007 12:16 pm

Re: NLMeans denoise filter

Post by BradleyS »

Conventional wisdom is that one must denoise before everything else. In practice, our combing detection works fine on noisy sources, and denoising with an averaging filter like NLMeans could actually make things worse. It feels wrong to me to say we don't denoise first, but we don't and it works great. So conventional wisdom is dated.

Detelecine > Comb detect > Decomb/Deinterlace > Denoise

HandBrake's version of NLMeans is not identical to any other implementation. It is based on Dr. Farin's helpful ffmpeg implementation. I've made changes, additions, and what I believe to be some improvements.

HandBrake's output is progressive. All displays going forward are and will be progressive, and software deinterlacers like ours are better than most displays/TVs. So you may want to Decomb/Deinterlace with HandBrake rather than attempt to produce interlaced video (which *will* be deinterlaced later on playback!). Decomb with default settings is usually fine, EEDI2 is sometimes higher quality but slow, and bob is good if you want to preserve smooth motion for that 6 o'clock news look. :)
Registered55
Enlightened
Posts: 121
Joined: Thu Apr 14, 2011 4:04 pm

Re: NLMeans denoise filter

Post by Registered55 »

EEDI2 Is extremely slow.... but one day in hope we may get something like a yadifmod - NNEDI 3 combo "Fast, efficient and great quality for the speed that it runs at"

Also it should be noted that there doesn't seem to be any (good) anti-bobbing filter used in handbrake to fix the awful issue that comes with all "BOB-bing" techniques.

It's just my view of course, but the bobbing effect more than outweighs any benefits that handbrakes bob deinterlacer gives.... but in fairness the BOB deinterlacing technique was created a very long time ago....LOL. It would be great however if there was some kind of anti-bobbing in handbrake to help reduce the effect of the BOB deinterlacer.

note... don't use EEDI2 with NLMeans unless you have a really really strong CPU...ouch.
Djfe
Bright Spark User
Posts: 178
Joined: Tue May 13, 2014 8:01 pm

Re: NLMeans denoise filter

Post by Djfe »

one thing I've wondered for a long time:
why would you want to use decombing instead of deinterlacing if you know the source is interlaced?
Post Reply