how to increase encoding quality

HandBrake for Mac support
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

how to increase encoding quality

Post by kvaruni » Tue Apr 06, 2010 2:48 pm

In the last few months, I have been encoding quite a number of BluRay sources. This works like a charm, but i am not entirely happy with the results. Let me first say that I am happy with roughly 95% of the encode. It is only in extreme cases that I see a bit too much blocking than I can tolerate for a HD source. So what is the problem?

This is how my encodes turn up: http://www.glowfoto.com/viewimage.php?y ... 6&srv=img6
and this is how the original looks: http://www.glowfoto.com/viewimage.php?y ... 1&srv=img6
(note that the contrast in these still images was lowered artificially to better expose the problem areas)

The wall, the floor and the green outdoors all suffer from the problem. This is clearly the result of compression (fun fact: the PNG of the encode is 4x smaller), but the thing that bothers me is that I can't seem to get rid of this blocking, even when upping RF to an insanely high value of 1 and turning everything off (no mbtree, no psy-rd, no-fast-pski, no-dct-decimate, heck, basically nothing but a single reference frame). The best results thus far are with an aq-strength set to 1 and deadzone-inter/intra set to 0. but even then, the results are not satisfactory. Are there any x264 options that can be used to lessen this effect? I am willing to let the file size increase with 100, even 200% and I do not care what the encoding speed is. Then again, I am not entirely sure if it can be done with only such a "moderate" file size increase, but any help is as always greatly appreciated!

mduell
Veteran User
Posts: 6674
Joined: Sat Apr 21, 2007 8:54 pm

Re: how to increase encoding quality

Post by mduell » Tue Apr 06, 2010 4:33 pm

And your encoding log is where?

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Tue Apr 06, 2010 6:14 pm

mduell wrote:And your encoding log is where?
Take whatever default setting you want. Or remove all advanced settings to resort to the defaults in x264. Make your pick. This is not a question of "I have a problem with my encode". This is more a question of "what specific x264 advanced options can be used to improve the quality in general, beyond those that are available in the GUI of handbrake".

dynaflash
Veteran User
Posts: 3820
Joined: Thu Nov 02, 2006 8:19 pm

Re: how to increase encoding quality

Post by dynaflash » Tue Apr 06, 2010 7:10 pm

kvaruni wrote:This is more a question of "what specific x264 advanced options can be used to improve the quality in general, beyond those that are available in the GUI of handbrake".
They all can be. The macgui has widgets for the most popular settings. The advanced option string in the text field below can take whatever options x264 has to offer. No matter how obscure (ie. you can play til your hearts content).

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Tue Apr 06, 2010 8:15 pm

dynaflash wrote:
kvaruni wrote:This is more a question of "what specific x264 advanced options can be used to improve the quality in general, beyond those that are available in the GUI of handbrake".
They all can be. The macgui has widgets for the most popular settings. The advanced option string in the text field below can take whatever options x264 has to offer. No matter how obscure (ie. you can play til your hearts content).
Indeed, I agree. I do know my way around most of the advanced settings, but I am at a loss with respect to this particular problem. I have seen this blockiness appear in nearly all my encodes working with different settings and from various sources (TV shows on DVD right up to BluRay). I can imagine that no at least some of you have encountered such compression artifacts and found them bothersome. So, how can you best alleviate this specific kind of compression "problems"? My best guesses were to turn off mbtree, increase the RF and play with aq-strength as well as deadzone settings and essentially only these last adjustments made any noticeable difference. So, any other options I can try? In fact, I would be happy if I could create a file with Blu-Ray quality at 720p instead of 1080p, regardless of file size ...

creamyhorror
Regular User
Posts: 134
Joined: Mon Jan 04, 2010 2:00 pm

Re: how to increase encoding quality

Post by creamyhorror » Wed Apr 07, 2010 1:59 am

I can't see your screenshots. Upload them to another site please.

There's no way you're seeing real blockiness in such high-bitrate encodes. I expect it's a decoder error/glitch, like weightp=2 incompatibility. (Or it could be the usual blocking in dark regions, but I doubt it...)

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Wed Apr 07, 2010 7:16 am

creamyhorror wrote:I can't see your screenshots. Upload them to another site please.

There's no way you're seeing real blockiness in such high-bitrate encodes. I expect it's a decoder error/glitch, like weightp=2 incompatibility. (Or it could be the usual blocking in dark regions, but I doubt it...)
Indeed, this does look like a weightp problem. I have seen this kind of problems creep up on the AppleTV and they look exactly the same. Then again ... this was encoded with weightp=0 ... I cross-checked my findings in both VLC and Quicktime, which both look terrible, and only mplayer displays them somewhat better (though this only seems to be because the colors and contrast are different, indeed, you can obtain the very same image by modifying the contrast and saturation of the picture). I redid the screenshots and uploaded them here:
http://users.telenet.be/kvaruni/good.png
http://users.telenet.be/kvaruni/bad.png

The Activity Log of the encode from which I took the frame is as follows:

Code: Select all

HandBrake Activity Log for /Users/kimbauters/Desktop/Bolt test 7.m4v: 2010-04-07 08:47:06 +0200
Handbrake Version: svn3197 x86_64 (2010040501)

[08:47:06] macgui: Rip: Pending queue count is 0
[08:47:06] macgui: scanning specifically for title: 1
[08:47:06] macgui: performNewQueueScan currentQueueEncodeIndex is: 6
[08:47:06] hb_scan: path=/Volumes/Drop Drive/Blu-Ray/Bolt.mkv, title_index=1
[08:47:06] scan: trying to open with libdvdread
libdvdread: Using libdvdcss version 1.2.10 for DVD access
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
GUI ERROR dialog: dvd: ifoOpen failed
GUI ERROR dialog: dvd: ifoOpen failed
Input #0, matroska, from '/Volumes/Drop Drive/Blu-Ray/Bolt.mkv':
  Duration: 01:36:21.85, start: 0.000000, bitrate: N/A
    Stream #0.0(eng): Video: h264, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 23.98 tbr, 1k tbn, 47.95 tbc
    Stream #0.1(eng): Audio: dca, 48000 Hz, 6 channels, s16
    Stream #0.2(nld): Audio: dca, 48000 Hz, 6 channels, s16
  Metadata
    original_writing_app: MakeMKV v1.4.9 beta darwin(x86-release)
[08:47:06] scan: decoding previews for title 1
[08:47:06] scan: audio 0x1: DCA, rate=48000Hz, bitrate=1536000 English (DTS) (5.1 ch)
[08:47:06] scan: audio 0x2: DCA, rate=48000Hz, bitrate=1536000 Nederlands (DTS) (5.1 ch)
[08:47:08] scan: 10 previews, 1920x1080, 23.976 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 1:1
[08:47:08] scan: title (0) job->width:1920, job->height:1072
[08:47:08] libhb: scan thread found 1 valid title(s)
[08:47:08] macgui: ScanDone state received from fQueueEncodeLibhb
[08:47:08] macgui: Preset: Custom
[08:47:08] macgui: processNewQueueEncode number of passes expected is: 1
[08:47:08] macgui: Start / Stop set to chapters
[08:47:08] macgui: prepareJob exiting
[08:47:08] 1 job(s) to process
[08:47:08] starting job
[08:47:08] sync: expecting 9078 video frames
[08:47:08] job configuration:
[08:47:08]  * source
[08:47:08]    + /Volumes/Drop Drive/Blu-Ray/Bolt.mkv
[08:47:08]    + title 1, chapter(s) 12 to 12
[08:47:08]    + container: matroska
[08:47:08]  * destination
[08:47:08]    + /Users/kimbauters/Desktop/Bolt test 7.m4v
[08:47:08]    + container: MPEG-4 (.mp4 and .m4v)
[08:47:08]      + 64-bit formatting
[08:47:08]  * video track
[08:47:08]    + decoder: h264
[08:47:08]    + frame rate: same as source (around 23.976 fps)
[08:47:08]    + dimensions: 1920 * 1080 -> 1280 * 720, crop 0/0/0/0
[08:47:08]    + encoder: x264
[08:47:08]      + options: level=31:ref=4:mixed-refs=1:bframes=3:b-adapt=2:direct=auto:weightb=1:b-pyramid=1:me=hex:subq=7:analyse=i8x8,i4x4:8x8dct=1:cabac=1:no-fast-pskip=0:no-dct-decimate=0:mbtree=1:weightp=0:ipratio=1.1:pbratio=1.1:aq-strength=1:deadzone-intra=10:deadzone-inter=10:qcomp=0.7:trellis=0:vbv-maxrate=10000:vbv-bufsize=2500:no-deblock=1
[08:47:08]      + quality: 20.00 (RF)
[08:47:08]  * audio track 0
[08:47:08]    + decoder: English (DTS) (5.1 ch) (track 1, id 1)
[08:47:08]      + bitrate: 1536 kbps, samplerate: 48000 Hz
[08:47:08]    + mixdown: Dolby Pro Logic II
[08:47:08]    + encoder: ca_aac
[08:47:08]      + bitrate: 128 kbps, samplerate: 44100 Hz
[08:47:08] encx264: keyint-min: 24, keyint-max: 240
[08:47:08] encx264: Encoding at constant RF 20.000000
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
x264 [info]: profile High, level 3.1
[08:47:14] reader: first SCR 334650510 id 0 DTS 334650510
Stream with high frequencies VQ coding
[08:48:39] macgui: Rip: Pending queue count is 0
[08:48:40] reader: done. 1 scr changes
[08:48:40] macgui: cancelling current job and stopping the queue
[08:48:40] sync: got 926 frames, 9078 expected
[08:48:40] work: average encoding speed for job is 10.837111 fps
[08:48:40] h264-decoder done: 938 frames, 0 decoder errors, 0 drops
[08:48:40] render: lost time: 0 (0 frames)
[08:48:40] render: gained time: 0 (0 frames) (0 not accounted for)
x264 [info]: frame I:6     Avg QP:15.50  size: 45881  PSNR Mean Y:51.03 U:53.15 V:51.97 Avg:51.47 Global:51.12
x264 [info]: frame P:287   Avg QP:16.91  size: 10787  PSNR Mean Y:50.25 U:52.89 V:51.21 Avg:50.73 Global:50.53
x264 [info]: frame B:567   Avg QP:19.76  size:  3357  PSNR Mean Y:50.15 U:53.11 V:51.28 Avg:50.69 Global:50.50
x264 [info]: consecutive B-frames:  2.1%  2.8% 75.9% 19.2%
x264 [info]: mb I  I16..4: 14.6% 63.9% 21.5%
x264 [info]: mb P  I16..4:  3.8%  6.5%  1.0%  P16..4: 56.8%  0.0%  0.0%  0.0%  0.0%    skip:31.9%
x264 [info]: mb B  I16..4:  0.5%  0.5%  0.0%  B16..8: 35.9%  0.0%  0.0%  direct: 2.5%  skip:60.6%  L0:40.2% L1:55.8% BI: 4.0%
x264 [info]: 8x8 transform intra:57.3% inter:63.8%
x264 [info]: direct mvs  spatial:99.5% temporal:0.5%
x264 [info]: coded y,uvDC,uvAC intra: 39.1% 50.0% 24.3% inter: 6.9% 7.4% 0.3%
x264 [info]: i16 v,h,dc,p: 37% 30%  7% 27%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 25% 29%  4%  5%  5%  7%  5%  5%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 30% 27%  3%  5%  4%  5%  3%  3%
x264 [info]: ref P L0: 60.7% 20.9% 13.2%  5.2%
x264 [info]: ref B L0: 77.1% 18.0%  4.8%
x264 [info]: ref B L1: 92.2%  7.8%
x264 [info]: SSIM Mean Y:0.9915767
x264 [info]: PSNR Mean Y:50.193 U:53.039 V:51.262 Avg:50.709 Global:50.510 kb/s:1113.04
[08:48:40] mux: track 0, 855 frames, 5265249 bytes, 1178.43 kbps, fifo 8
[08:48:40] mux: track 1, 1539 frames, 589906 bytes, 132.03 kbps, fifo 256
[08:48:41] libhb: work result = 0
If you are wondering, removing level and the vbv options does not affect the output, no-deblock was an experiment of mine but does not make a real difference, grain retention through deadzone provides some slight improvements, mbtree off just blows up the file size, no-fast-pskip and no-dct-decimate seem to have little effect, varying qcomp slightly makes no real difference, analysing all blocks just bumps up the encode time and likewise for the motion estimation method. Finally, lowering the RF value has no real impact in these scene.

mkelley
Bright Spark User
Posts: 389
Joined: Fri Dec 25, 2009 2:00 am

Re: how to increase encoding quality

Post by mkelley » Wed Apr 07, 2010 10:13 am

Just to confirm, played on my WDLive I have seen zero blockiness or any other problems even with my blu-ray sources (which I use the Handbrake High Preset now with no changes other than to output to MKV and passthrough the DTS or AC3 audio).

It's not Handbrake that's causing the problem.

User avatar
Rodeo
HandBrake Team
Posts: 12203
Joined: Tue Mar 03, 2009 8:55 pm

Re: how to increase encoding quality

Post by Rodeo » Wed Apr 07, 2010 12:57 pm

kvaruni wrote:The Activity Log of the encode from which I took the frame is as follows:

Code: Select all

[08:47:08]      + options: level=31:ref=4:mixed-refs=1:bframes=3:b-adapt=2:direct=auto:weightb=1:b-pyramid=1:me=hex:subq=7:analyse=i8x8,i4x4:8x8dct=1:cabac=1:no-fast-pskip=0:no-dct-decimate=0:mbtree=1:weightp=0:ipratio=1.1:pbratio=1.1:aq-strength=1:deadzone-intra=10:deadzone-inter=10:qcomp=0.7:trellis=0:vbv-maxrate=10000:vbv-bufsize=2500:no-deblock=1
You've got to be freaking kidding… you start a topic with the following:
kvaruni wrote:It is only in extreme cases that I see a bit too much blocking than I can tolerate for a HD source. So what is the problem?
Only to post the log of an encode where you've disabled the in-loop deblocking filter?

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Wed Apr 07, 2010 1:22 pm

mkelley wrote:Just to confirm, played on my WDLive I have seen zero blockiness or any other problems even with my blu-ray sources (which I use the Handbrake High Preset now with no changes other than to output to MKV and passthrough the DTS or AC3 audio).

It's not Handbrake that's causing the problem.
The more tests I run, the more I agree with you. I end up with files that are bigger than the original Blu-Ray, so even in the worst case I should already have seen some improvement :).
Rodeo wrote:Only to post the log of an encode where you've disabled the in-loop deblocking filter?
Just a test, you can't know if you haven't tried it. Then again, most deblocking filters apply some form of smoothing and the problem is in such smooth areas. It was a fair shot to try. Regardless, results are the same.

nightstrm
Veteran User
Posts: 1887
Joined: Fri Mar 23, 2007 5:43 am

Re: how to increase encoding quality

Post by nightstrm » Wed Apr 07, 2010 1:49 pm

kvaruni wrote: The more tests I run, the more I agree with you. I end up with files that are bigger than the original Blu-Ray, so even in the worst case I should already have seen some improvement :).
That's a joke, right? :shock:

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Thu Apr 08, 2010 5:26 pm

So, I got the settings nailed. It was in the settings after all. The input was just fine, the decoder hardly matters. Now, for this particular source (animation) the solutions seems to be to allow rather aggressive psychovisual trellis (value of 0.5) as well as upping qcomp (I chose 1 as the maximum). The result is obviously a far bigger file (+ 60%, 2.1GB for Bolt), but all my gripes are resolved. Furthermore, I stay within Main profile and don't move up to High, allowing me some headroom for further reducing the file size by sacrificing encoding speed. Any modification of deblocking was not needed and I removed the deadzone tweaks as they were obsolete because of the trellis that is now enabled. I also got rid of weightp, just to get a file that was compatible with the AppleTV and all other devices I own and want to play fulltime movies on.

creamyhorror
Regular User
Posts: 134
Joined: Mon Jan 04, 2010 2:00 pm

Re: how to increase encoding quality

Post by creamyhorror » Thu Apr 08, 2010 8:30 pm

The settings you posted:

Code: Select all

+ options: level=31:ref=4:mixed-refs=1:bframes=3:b-adapt=2:direct=auto:weightb=1:b-pyramid=1:me=hex:subq=7:analyse=i8x8,i4x4:8x8dct=1:cabac=1:no-fast-pskip=0:no-dct-decimate=0:mbtree=1:weightp=0:ipratio=1.1:pbratio=1.1:aq-strength=1:deadzone-intra=10:deadzone-inter=10:qcomp=0.7:trellis=0:vbv-maxrate=10000:vbv-bufsize=2500:no-deblock=1
are just weird. analyse=i8x8,i4x4? ipratio=1.1? That's got to have some strange effects. Then you set vbv-bufsize to 2500, which is limiting your local bitrate to 2500...not a good idea for 720p. And then there's no-deblock. Your log indicates a pretty low bitrate (1113kbps), but only over ~900 frames, and I don't know if it's because of deadzone settings keeping bitrate low or if it's because of the source itself.

Since there are too many issues here, I don't think there's any conclusion to be drawn for the rest of us. If you found you liked qcomp=1, you probably just don't like the effect of mbtree, since qcomp=1 basically turns it off and uses constant quantization.

kvaruni
Posts: 56
Joined: Tue Mar 24, 2009 1:12 pm

Re: how to increase encoding quality

Post by kvaruni » Fri Apr 09, 2010 6:54 am

creamyhorror wrote:The settings you posted:

Code: Select all

+ options: level=31:ref=4:mixed-refs=1:bframes=3:b-adapt=2:direct=auto:weightb=1:b-pyramid=1:me=hex:subq=7:analyse=i8x8,i4x4:8x8dct=1:cabac=1:no-fast-pskip=0:no-dct-decimate=0:mbtree=1:weightp=0:ipratio=1.1:pbratio=1.1:aq-strength=1:deadzone-intra=10:deadzone-inter=10:qcomp=0.7:trellis=0:vbv-maxrate=10000:vbv-bufsize=2500:no-deblock=1
are just weird. analyse=i8x8,i4x4? ipratio=1.1? That's got to have some strange effects. Then you set vbv-bufsize to 2500, which is limiting your local bitrate to 2500...not a good idea for 720p. And then there's no-deblock. Your log indicates a pretty low bitrate (1113kbps), but only over ~900 frames, and I don't know if it's because of deadzone settings keeping bitrate low or if it's because of the source itself.

Since there are too many issues here, I don't think there's any conclusion to be drawn for the rest of us. If you found you liked qcomp=1, you probably just don't like the effect of mbtree, since qcomp=1 basically turns it off and uses constant quantization.
Turning off mbtree by itself doesn't do the trick. It has to be in combination with a fairly high psychovisual trellis. The settings as they are may seem to be a bit strange, but remember that this is just one test set, the one I happened to be working on when I read the replies. To answer some of your questions: analyse is chosen to have some analysis, though with minimum impact on encoding speed. Ipratio was chosen at that time following the megui guide for grainy content, since this was the problem I was facing. And yes, the results are just fine. The vbv-maxrate and vbv-bufsize are cherry-picked values that are just about the maximum that my AppleTV will take for any Blu-Ray encodings, including some very tough Planet Earth material. I have seen other solutions and other values, but only with these settings in place am I sure that Blu-Ray content will run just fine on the AppleTV. Surprisingly, I have found that removing it rarely has any effect on the quality of the final encode. It is only an upper bound in the case of some very hard Blu-Ray source material. Yes, I would like Apple to update the AppleTV so that it supports each and every settings in the High profile, but for the time being I'm just out of luck. And as indicated, no-deblock was just a test. Once again, the settings really don't matter.If I pick the standard High profile setting I end up with just the same blockiness. For some reason, it just doesn't retain enough detail without the trellis/qcomp. As for the low bitrate and the limited frames, this is due to the source. No worries there.

I redid the tests with no advanced settings whatsoever. Same result (and please tell me any issues you may have with using the defaults :)). Bump this up with a high qcomp and a high psychovisual trellis and the result is suddenly just fine. Turn it off again and all large areas with subtle gradients become terrible to look at, in my very own subjective opinion, obviously :). I stick to the standard settings for any SD source, but I just find that there is too little detail retention when encoding a HD source and I can most definitely live with a file that is twice the size if I don't get interrupted during a movie by some ugly wall, blocky smoke or layered cloud :).

creamyhorror
Regular User
Posts: 134
Joined: Mon Jan 04, 2010 2:00 pm

Re: how to increase encoding quality

Post by creamyhorror » Fri Apr 09, 2010 2:06 pm

The problems you're seeing aren't clear to me (or others, I suspect) from the screenshots you posted, because you disabled deblocking and because there's a levels issue that's preventing a direct comparison of the quality (your playback program's levels settings are wrong; I hope you're not using VLC). The lack of deblocking clearly makes a difference in the screenshots since there's very obvious blocking in the encode. Plus you didn't resize the pics to the same size.

Basically, without seeing the actual problem, we can't address it.

If you can post 3 screenshots -
(1) source
(2) encode with your special settings
(3) encode with the same settings as (2) except for the psy-trellis and qcomp values and with the same bitrate as (2) via 2-pass

- then it would go a long way to showing us the actual effect of your settings. The bitrates need to match and the only settings differences between (2) and (3) should be psy-trellis and qcomp, else the comparison is pretty worthless.

(Given that your settings result in a much higher bitrate, it's extremely probable that the bitrate boost is what's making most of the difference. So we have to eliminate that possibility.)

Post Reply