Need help reducing noise in Looney Tunes DVD encodes

[Gooch]

I'm re-encoding my Looney Tunes cartoons, and I'm really having difficulties with Hanbrake's NLMeans denoise filter. I've tried the ultralight and light denoise settings with the Animation Tune , high profile 3.1, Animation tune, RF 18-20. In a lot of cases, the file size is still over 200 MB for a 6 minute cartoon.

I want to recreate the superb noise reduction shown in the first screenshot. It has surprisingly little loss of detail compared to the Handbrake encodes which have overly blurred lines and detail while still leaving lots of mosquito (?) noise and artifacts along edges and lines. With heavy denoise, there is quite a lot of detail lost (e.g., the bubbles).

DVD screenshot showing noise


Desired denoise result of "Hare Remover"


DVD encode - HB Utralight denoise (animation tune)


DVD encode - HB Light denoise (animation tune)


DVD encode - HB Medium denoise (animation tune)


DVD encode - HB Strong denoise (animation tune)


As you can see, the HB denoise filter leaves a lot of blotchiness which the first encode doesn't have.

I would really like to be able to get Handbrake's denoise filter to produce results like in the first photo, but I don't know how I can do this. Handbrake only has a few denoise presets. Is it possible to recreate the denoise settings using the encoding information from the MediaInfo report?

Writing library : x264 core 66 r1115 11863ac
Encoding settings : cabac=1 / ref=16 / deblock=1:-2:-1 / analyse=0x3:0x133 / me=tesa / subme=9 / psy_rd=1.5:0.0 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=5 / nr=0 / decimate=1 / mbaff=0 / bframes=12 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40(pre) / rc=2pass / bitrate=1966 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00

[musicvid]

You know, that's 35mm film stock and most people would like to keep the grain.
That said, you may want to run some trials with NLMeans at various settings, and leave denoise off.

I know it was unintentional, but read the forum rules wrt to the sources of your screenshots.

[Gooch]

You know, that's 35mm film stock and most people would like to keep the grain.
That said, you may want to run some trials with NLMeans at various settings, and leave denoise off.

Yes, I agree for the most part, but the file sizes I'm getting without denoise on are quite large. That's why I really like the noise reduction as shown in the first photo; it retains a lot of fine detail while taking out much of the unattractive jaggy noise.

How do I use NLMeans with denoise off? I'm only familiar with using Handbrake's presets in the GUI. Do I enter something in the "Additional Options" field under the Video section?

[musicvid]

It's in the Denoise dropdown.

[nhyone]

Is it possible to show how the original looks like?

I don't have much experience with NLMeans, but if you use regular denoise, I would use ultralight spatial and strong temporal settings, such as 1:1:1:5:5:5 or even 2:2:2:7:7:7 (the highest I would go). [HandBrake 0.10.x requires 6 numbers.]

So far, something like that works well for me for grain -- it gets caught by the temporal filter since it shifts around every frame.

[Gooch]

Is it possible to show how the original looks like?

I don't have much experience with NLMeans, but if you use regular denoise, I would use ultralight spatial and strong temporal settings, such as 1:1:1:5:5:5 or even 2:2:2:7:7:7 (the highest I would go). [HandBrake 0.10.x requires 6 numbers.]

So far, something like that works well for me for grain -- it gets caught by the temporal filter since it shifts around every frame.

Hmmm, I entered 1:1:1:5:5:5 into Handbrake's NLMeans custom field and ran a test. I didn't notice any difference. It looked as if no denoise was applied at all.

[nhyone]

Not NLMeans, but the regular denoise.

Choose hqdn3d, then custom preset, then enter the values.

[Gooch]

Is it possible to show how the original looks like?

I don't have much experience with NLMeans, but if you use regular denoise, I would use ultralight spatial and strong temporal settings, such as 1:1:1:5:5:5 or even 2:2:2:7:7:7 (the highest I would go). [HandBrake 0.10.x requires 6 numbers.]

So far, something like that works well for me for grain -- it gets caught by the temporal filter since it shifts around every frame.

I tried your suggested settings. The denoise effect was very subtle, but sadly not what I'm looking for.

I think I'll just encode the rest of my DVDs with denoise off. Whoever encoded the file I used for comparison is obviously very knowledgeable about video encoding and denoise. I can't come close to approximating it with Handbrake's denoise presets. I was hoping someone could "reverse engineer" the encoding settings from the MediaInfo dump I provided, but I guess not. I'm not even sure that Handbrake can match the results of the software the other person used (Writing application : mkvmerge; v2.5.2 ('Stranger in your Soul') built on Feb 28 2009 18:58:17; Writing library : libebml v0.7.7 + libmatroska v0.8.1)

I'm might give it one more shot. If I succeed, I'll post in the thread. A good tutorial on how to use the custom settings for NLMeans would be very helpful.

[musicvid]

What confuses me so much, is that in spite of a standing request, you've never posted an original screenshot and log of your DVD rip, only those of a pirated download and some possibly misapplied denoise attempts in Handbrake.

How about your personal source DVD? What's it look like?
Without a stable reference, all you've done is to present your peers with a moving target.
Is that what you intended?

[rollin_eng]

What confuses me so much, is that in spite of a standing request, you've never posted an original screenshot and log of your DVD rip, only those of a pirated download and some possibly misapplied denoise attempts in Handbrake.

How about your personal source DVD? What's it look like?
Without a stable reference, all you've done is to present your peers with a moving target.
Is that what you intended?

Looks like the link in the OP has been updated to show the DVD source.

[nhyone]

Thanks to Gooch for providing me a 3-min clip so that I was able to try out various settings.

The encoding settings is informative, but the most important setting, as far as size is concerned, is bitrate=1966. It uses 1.966 Mbps.

I got 5.094 Mbps for my first attempt (log at the end of this post), compared to the original's 5.278 Mbps. Perhaps the CRF was too low?

Code: Select all

CRF  bitrate
org  5.278
20   5.094
22   4.206
24   3.298
26   2.263
27   1.781
28   1.348

CRF 28 was when it crossed my visual threshold, so I would go with either 26 or 27. With CRF 27, the bitrate is already lower than the reference -- assuming both have the same average bitrate -- and it still has most of the grain!

Next is to denoise it. As mentioned, hqdn3d denoise does not work. Looking at the failed attempt, my take is the grain is (i) attenuated [due to insufficient DVD bitrate], (ii) aliased and coarser [due to SD res] and (iii) mixed with MPEG-2 artifacts.

But NLMeans denoise can work. After a few tests, I settled on 6:0.5:3:5:6:0. What it means:

Strength 6. 4 was a bit too weak, and 8 was too strong.
Origin weight 0.5. My default value.
Patch size 3. This cannot be too high for SD res. 5 made it too soft.
Search range 5. Just search nearby.
Frames 6. I usually use a max of 1/4s for animation.
Pre-filtering mode 0. Doesn't seem to have much effect, so I don't use it.

The result, at CRF 27, uses 0.705 Mbps. The video still has a bit of noise, which helps to hide the MPEG-2 artifacts. It is good enough for me, but I would still keep the source -- in case we found how to encode better in the future!


Encoding log for my first attempt:

Code: Select all

[11:02:10] job configuration:
[11:02:10]  * source
[11:02:10]    + looney_tunes_sample.mov
[11:02:10]    + title 1, chapter(s) 1 to 1
[11:02:10]    + container: mov,mp4,m4a,3gp,3g2,mj2
[11:02:10]    + data rate: 5278 kbps
[11:02:10]  * destination
[11:02:10]    + crf20.mkv
[11:02:10]    + container: Matroska (libavformat)
[11:02:10]  * video track
[11:02:10]    + decoder: mpeg2video
[11:02:10]      + bitrate 9800 kbps
[11:02:10]    + filters
[11:02:10]      + Decomb (default settings)
[11:02:10]      + Framerate Shaper (0:27000000:1080000)
[11:02:10]        + frame rate: same as source (around 25.000 fps)
[11:02:10]      + Crop and Scale (720:576:0:0:0:0)
[11:02:10]        + source: 720 * 576, crop (0/0/0/0): 720 * 576, scale: 720 * 576
[11:02:10]    + custom anamorphic
[11:02:10]      + storage dimensions: 720 * 576, mod 2
[11:02:10]      + pixel aspect ratio: 16 / 15
[11:02:10]      + display dimensions: 768 * 576
[11:02:10]    + encoder: H.264 (libx264)
[11:02:10]      + preset:  veryslow
[11:02:10]      + options: aq-strength=0.7:psnr=0:ssim=0:ref=4:bframes=6
[11:02:10]      + quality: 20.00 (RF)
[11:02:10] reader: first SCR -17190 id 0x0 DTS -17190
[11:02:10] encx264: encoding at constant RF 20.000000
[11:02:10] encx264: unparsed options: aq-strength=0.70:psnr=0:ssim=0:ref=4:bframes=6:b-adapt=2:direct=auto:analyse=all:me=umh:merange=24:subme=10:trellis=2:rc-lookahead=60
x264 [info]: using SAR=16/15
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x264 [info]: profile High, level 3.0
[11:02:10] sync: first pts is 3600
[11:05:34] reader: done. 1 scr changes
[11:05:41] work: average encoding speed for job is 21.806171 fps
[11:05:41] mux: track 0, 4492 frames, 114420082 bytes, 5094.39 kbps, fifo 512
[11:05:41] sync: got 4493 frames, 4493 expected
[11:05:41] decomb: deinterlaced 2 | blended 39 | unfiltered 4451 | total 4492
[11:05:41] render: lost time: 0 (0 frames)
[11:05:41] render: gained time: 0 (0 frames) (0 not accounted for)
[11:05:41] mpeg2video-decoder done: 4493 frames, 0 decoder errors, 0 drops
x264 [info]: frame I:34    Avg QP:20.58  size: 53045
x264 [info]: frame P:1165  Avg QP:23.44  size: 33467
x264 [info]: frame B:3293  Avg QP:25.11  size: 22359
x264 [info]: consecutive B-frames:  2.3%  5.4% 15.8% 37.1% 20.0% 16.0%  3.3%
x264 [info]: mb I  I16..4:  2.5% 94.7%  2.8%
x264 [info]: mb P  I16..4:  0.7% 59.5%  1.2%  P16..4: 17.7% 10.5%  6.4%  0.2%  0.0%    skip: 3.6%
x264 [info]: mb B  I16..4:  0.2% 39.2%  0.4%  B16..8: 26.1% 10.0%  3.2%  direct: 9.8%  skip:11.1%  L0:36.8% L1:35.2% BI:28.0%
x264 [info]: 8x8 transform intra:97.9% inter:91.9%
x264 [info]: direct mvs  spatial:99.7% temporal:0.3%
x264 [info]: coded y,uvDC,uvAC intra: 95.1% 91.9% 48.3% inter: 59.5% 62.1% 20.0%
x264 [info]: i16 v,h,dc,p: 16% 30% 45%  8%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  9%  9% 69%  2%  2%  2%  2%  2%  3%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 11%  9% 10% 12% 14% 10% 10%  9%
x264 [info]: i8c dc,h,v,p: 36% 33% 24%  7%
x264 [info]: Weighted P-Frames: Y:40.1% UV:35.9%
x264 [info]: ref P L0: 46.2% 16.2% 19.2% 12.3%  5.2%  0.9%
x264 [info]: ref B L0: 82.4% 14.0%  3.6%
x264 [info]: ref B L1: 90.2%  9.8%
x264 [info]: kb/s:5094.46
[11:05:41] libhb: work result = 0

[arcuser]

Are you encoding these from the original DVD source?

Are you sure that whoever encoded the "Desired denoise result" screenshot did not take them from the Blu-Ray version?

[Gooch]

Are you encoding these from the original DVD source?

Are you sure that whoever encoded the "Desired denoise result" screenshot did not take them from the Blu-Ray version?

As far as I know, there is no Blu-Ray version of this particular cartoon, "Hare Remover."

nyhone's custom noise filter settings works very well. I'm happy with it. It doesn't quite match the "desired" denoise results in the screenshot I posted, but it comes decently close.

[stryfewalker]

I fought with inflated file sizes with the old "Scooby Doo, Where Are you" cartoon dvd series as it was fairly noisy. I didn't want to apply heavy denoising and all the light stuff was barely making a difference so I finally settled on using two pass encoding at a fixed bit rate. I know you are focused on quality denoising, but just thought I'd mention this for anyone reading this thread that might be more concerned with large file sizes.

I know two pass encoding is looked down on here but for encodes to mainly be watched by my kids on small tablet screens, it was a very serviceable solution. Frankly I didn't notice much of a difference in picture quality of 1500 bitrate two-pass vs the CRF size (17-19ish) that was anywhere from 30-50% larger.

[musicvid]

Well, that is a bit of a sledgehammer approach; the thing I would look at is high motion areas at 1500 Kbps.
If it looks good to you, it speaks well for the encoder.

[mduell]

I know two pass encoding is looked down on here but for encodes to mainly be watched by my kids on small tablet screens, it was a very serviceable solution. Frankly I didn't notice much of a difference in picture quality of 1500 bitrate two-pass vs the CRF size (17-19ish) that was anywhere from 30-50% larger.

You could use a higher CRF and get the same size...

[nhyone]

Frankly I didn't notice much of a difference in picture quality of 1500 bitrate two-pass vs the CRF size (17-19ish) that was anywhere from 30-50% larger.

The problem is that your CRF is too low.

It is usually recommended to use CRF 18-20 for transparent encoding, but there is an implicit assumption -- that the source is of sufficiently high quality. If the source is encoded with an equivalent CRF of, say, 26, then using CRF 20 just needlessly bloats the size. That is why sometimes we need to hunt for the "correct" CRF.

Also, grain that has been (poorly) encoded with MPEG-2 has been "optimized" for MPEG-2 encoding. This happens when the bitrate is not high enough. However, to H.264, it looks like details. Thus it needs to use a higher than expected bitrate at a given CRF.

[stryfewalker]

Sorry, didn't mean to alter the conversation on the thread. Basically, when it comes to mpeg2 to h.264 conversion I have never been that happy with the results of CRF with relation to file size. Anything above about 18-19-20 felt noticeable to me but the two pass approach seemed to offer a good balance of file size/quality. I don't mind the encodes taking nearly twice as long so that's not even an issue.

The two pass approach I just use for mpeg2/dvd sources (I hate those pesky mpeg2 coding used for the sd features on some blurays), with blurays I always use CRF values and am usually happy with 21-22.