Tune custom NLMeans parameters

Discuss encoding for devices and presets.
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Tune custom NLMeans parameters

Post by Nomis101 »

I have two BluRay movies which do have bad noise in it. It is a noise that flickers around in the movie very annoyingly (“dancing dots”). The result is, that the encoded movie has 20 GB in size (compared to 5 GB that my non-grainy cartoons normally do have).

So, I was testing different NLMeans presets, but even for Ultralight I see a loss in details. Light (and even medium) looks very blurry. I've also experimented with the old HQDN3D, which removes the “dancing dots” very nicely, while keeping the details, for

Code: Select all

y-spatial=1:cb-spatial=1:cr-spatial=1:y-temporal=8:cb-temporal=8:cr-temporal=8
But HQDN3D changes the colors of my movie a bit. Is it possible to bring HQDN3D parameters to NLMeans parameters? And how would my HQDN3D parameters would look like?

The docs says about NLMeans parameters:
Custom Format:
y-strength=y:y-origin-tune=y:y-patch-size=y:
y-range=y:y-frame-count=y:y-prefilter=y:
cb-strength=c:cb-origin-tune=c:cb-patch-size=c:
cb-range=c:cb-frame-count=c:cb-prefilter=c:
cr-strength=c:cr-origin-tune=c:cr-patch-size=c:
cr-range=c:cr-frame-count=c:cr-prefilter=c:
threads=t

Default:
y-strength=6:y-origin-tune=1:y-patch-size=7:
y-range=3:y-frame-count=2:y-prefilter=0:
cb-strength=6:cb-origin-tune=1:cb-patch-size=7:
cb-range=3:cb-frame-count=2:cb-prefilter=0
But, this looks very very complicated. Sure, I've read this thread: viewtopic.php?f=4&t=30135&p=139077 But I still don't quite understand what all those parameters do. And it would take a hell of a lot of time to try different numbers for all those parameters.
Could somebody suggest me numbers for the NLMeans parameters for my movie, which will remove the “dancing dots”, but preserve the details?


No denoise:
Image


Custom HQDN3D:
Image


NLMeans - Ultralight - Film
Image


NLMeans - Light - Film
Image


By the way, I quite don't understand why the "Default" NLMeans syntax is much shorter then the "Custom Format"?

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

Looks like the images didn't come through.

If you can provide a short sample of your source, I can try to find optimal settings and explain what they do.

The reason the custom format is longer is that the parameters cascade. In YCbCr video (similar to YUV), we have a luminance channel (think black and white detail), and two chroma/color channels. When setting custom NLMeans parameters, the values cascade from Y to Cb to Cr, so if you set a strength for the Y channel only, it will be used for the chroma channels also; likewise, if you set a strength for Y and a different strength for Cb, the Cr channel will use the Cb setting. The official NLMeans presets all use the same settings for both chroma channels, so you're seeing HandBrake set only Cb which cascades to Cr also. Also, the official presets use the default thread count internal to NLMeans, so this is not set explicitly.

nhyone
Bright Spark User
Posts: 249
Joined: Fri Jul 24, 2015 4:13 am

Re: Tune custom NLMeans parameters

Post by nhyone »

Nomis101 wrote:
Sat Dec 07, 2019 10:27 pm
I've also experimented with the old HQDN3D, which removes the “dancing dots” very nicely, while keeping the details, for

Code: Select all

y-spatial=1:cb-spatial=1:cr-spatial=1:y-temporal=8:cb-temporal=8:cr-temporal=8
But HQDN3D changes the colors of my movie a bit.
You can set the cb and cr parameters to 0 to skip processing chroma channels.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

BradleyS wrote:
Sun Dec 08, 2019 6:46 am
Looks like the images didn't come through.
Maybe it works better with links.

No denoise:
http://polysom.verilite.de/tmp/Denoise-None.jpg

Custom HQDN3D:
http://polysom.verilite.de/tmp/Denoise-HQDN3D.jpg

NLMeans - Ultralight - Film
http://polysom.verilite.de/tmp/Denoise-Ultra.jpg

NLMeans - Light - Film
http://polysom.verilite.de/tmp/Denoise-Light.jpg


BradleyS wrote:
Sun Dec 08, 2019 6:46 am

If you can provide a short sample of your source, I can try to find optimal settings and explain what they do.
Thanks, here is a small sample: http://polysom.verilite.de/tmp/Sample.zip

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

Looks like film damage and dirt. Typically, this is cleaned up in a proper editor, but I might be able to find some NLMeans settings that help. I'd say the most annoying are the white spots that periodically appear for a single frame.

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

NLMeans alone won't deal with these film artifacts. Given how it works, it's just not going to find nearby spatial or temporal pixels similar enough to the source artifacts to give them enough weighting in the filtering process, even when tuning the origin weight. hqdn3d is more of a blunt instrument in this sense.

I've created some ffmpeg settings that seem to do better and may serve as a preprocessing pipeline for you. After running the source through ffmpeg with these settings, you can then use HandBrake on the result with the NLMeans and sharpening settings you like.

Code: Select all

ffmpeg -i input.mkv -filter_complex removegrain=m0=17,removegrain=m0=17,deflate=threshold0=50000,deflicker=size=8:mode=gm -crf 0 output.mkv
Removegrain mode 17's description is "Clips the pixel with the minimum and maximum of respectively the maximum and minimum of each pair of opposite neighbour pixels." This technique does a good job at minimizing stray pixels, shrinking and/or muting them somewhat. Running it twice seems to be a good balance between filtering and detail retention. There isn't a ton of detail in the sample so test this on more critical clips.

The deflate filter's description is "This filter replaces the pixel by the local(3x3) average by taking into account only values lower than the pixel." Now that we've shrunk most of the bright dots, deflate will attempt to remove the smallest ones via averaging.

Optionally, the deflicker filter can remove... well, flickering. I noticed a fair bit in the source. This setting might not be suitable for the entire source, but worked well on the sample. It works by measuring the brightness variations between frames and boosting/cutting the luminance of each to better match its neighbors. The effect is very subtle but pleasing.

You can use any quality metric you like for the output. CRF 0 is lossless for x264, so if you have a couple hundred gigs of space or more, this is your best bet. Otherwise, CRF 2 will reduce size considerably while being near-lossy (mathematically, not just visually). I would not go beyond CRF 10. CRF 2 and 10 also correspond to HandBrake's Production Max and Standard presets, respectively.

One option is to map and pass through the audio channels into the output and feed the entire result to HandBrake for NLMeans, sharpening, and all the rest. Another option is to only process the video (pass -an to disable audio) and use MKVToolNix or similar to create a new MKV with the processed video and original audio and any other tracks before feeding to HB. If you're only doing a couple this way, the latter option might be easier than sorting out audio on the ffmpeg command line.

Hopefully this helps. Let us know how you get on.

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

P.S. — If you need a current ffmpeg binary for macOS, there are prebuilt binaries here: https://evermeet.cx/ffmpeg/

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

Thanks BradleyS. This sounds like a time-consuming task. I will try your suggested way with the entire movie (will possibly take some days), and let you know. I have two movies with the same sort oft noise.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

P.S. - I regularly build my own custom made ffmpeg from source (with all the HandBrake patches applied).

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

So, I did some tests.

Code: Select all

ffmpeg -i input.mkv -filter_complex removegrain=m0=17,removegrain=m0=17,deflate=threshold0=50000,deflicker=size=8:mode=gm -crf 0 output.mkv
did remove the noise nicely, but I had a big loss in details. After removing one removegrain and the deflate part, I still had a little noise, but also most of the details. I was quite happy with the result. Then I noticed, that the deflicker filter has a bad side effect. If there is a change from a scene with light to a scene with not much light, then the deflicker filter makes the last frame of the foregoing scene suddenly very dark. So, I ended up with just:

Code: Select all

ffmpeg -i input.mkv -filter_complex removegrain=m0=17 -crf 4 -map 0:a -map 0:s output.mkv
This keeps most of the details and does reduce the noise in such a way, that HandBrake can handle the rest. But, in my test, I'm more happy with the results that HQDN3D

Code: Select all

y-spatial=1:cb-spatial=0:cr-spatial=0:y-temporal=7:cb-temporal=7:cr-temporal=7
does give me. The drawback is, that HQDN3D does not reduce the noise very good for moving objects. But I can live with that.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

Nomis101 wrote:
Mon Dec 09, 2019 4:23 pm
But, in my test, I'm more happy with the results that HQDN3D

Code: Select all

y-spatial=1:cb-spatial=0:cr-spatial=0:y-temporal=7:cb-temporal=7:cr-temporal=7
does give me.
No, I'm not, this was a false estimation.
I tested with different parts of the movie and a lot of different parameters for NLMeans. And overall

Code: Select all

y-strength=2.5:y-origin-tune=0.9:y-patch-size=3:y-range=4:y-frame-count=8:y-prefilter=1025:cb-strength=1:cb-origin-tune=0.8:cb-patch-size=3:cb-range=4:cb-frame-count=3:cb-prefilter=1025
gives me nice results. It reduces the noise quite good, but sadly also a bit of the details. It doesn't work so nice for the sample I've provided, but for most parts of the movie.
Which parameter could bring me back the details?

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

I wouldn't exceed 3 or so temporal frames for film content. Try reducing 8 to 3 and increasing the spatial search range.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

Thanks Bradley. What will happen, if temporal frames for film content will exceed 3? Will it introduce ghosting? I increased the spatial search range a bit, but it seems, this also massively reduced encoding speed. While changing the value for spatial search range, I noticed, that I did not used an odd number. So, I'm wondering if this had worked at the first place.

I now ended up with:

Code: Select all

y-strength=2.5:y-origin-tune=0.9:y-patch-size=3:y-range=5:y-frame-count=3:y-prefilter=1025:cb-strength=1:cb-origin-tune=0.8:cb-patch-size=3:cb-range=3:cb-frame-count=3:cb-prefilter=1025

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

Where an odd number is required, an even supplied value is decremented by 1.

More temporal frames can be very bad for chroma and introduce smearing since there's no compensation for movement. Luma, not as bad, but you won't get good matches either with high motion. 2-3 is usually good for film. The High Motion tune processes chroma spatially only to guarantee no ghosting/smearing.

You can estimate processing per frame by multiplying patch size * range. Multiplying again by frames seems reasonable with many frames can cause more memory pressure which can sometimes be detrimental to speed, so it's more of an estimate.

So with the settings above, you have 3*5*3 for luma, so every luma pixel processed requires analyzing 45 neighbors including the origin.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

Thanks for the info. Encoding time has increased from about 35 hours (without NLMeans) to around 49 hours (with NLMeans). So, tomorrow evening I will know how the complete movie looks like with denoising.

Nomis101
Posts: 33
Joined: Mon Dec 25, 2017 6:37 pm

Re: Tune custom NLMeans parameters

Post by Nomis101 »

Nomis101 wrote:
Tue Dec 17, 2019 10:36 am
So, tomorrow evening I will know how the complete movie looks like with denoising.
First movie with mentioned NLMeans parameters is done. And I'm quite impressed about HandBrakes NLMeans implementation. :) I haven't been very involved with denoising yet, but the movie does seem to have different kind of noises, depending on the scene. For some scenes denoising worked very excellent and for some scenes there is still some visible noise left. But overall, the movie looks much cleaner and crisper now. And, I had 50% reduction in file size.

User avatar
BradleyS
Moderator
Posts: 1769
Joined: Thu Aug 09, 2007 12:16 pm

Re: Tune custom NLMeans parameters

Post by BradleyS »

:D

Post Reply