status of HD encoding @ July 2009...?

[jingo_man_dan]

hi all

ok, as a background, i watched a number of threads about encoding dvd's (at best qualities) for the TV when i first bought the device, which was a v1.0 device probably circa mid-2007. at the time, with the help of legends such as cavalicious (apologies for any mispellings!) and dynaflash, et al, i decided upon a basic profile to use for my encodes. and that was the end of the tale. this was around Constant Quality of 70% and a bunch of options, including VBV Max rate, etc.

since this time, i have branched my source media, particularly into HD content (namely from HDTV or BluRay). in addition to this, apple has released further firmware updates, taking advantage of new aspects of h.264 and the GPU processing capabilities aswell, and Handbrake has also, independently, updated its version numbers too...

so here i am, mid-2009, and having read dozens of threads, my profile(s) are woefully out-of-date! i have updated the dvd profile based on the above legends "preferred" settings and have had good results, so case closed there ;o)

the main source of my information has been threads such as "ATV 2.x Advanced Settings (All Sources)" (http://forum.handbrake.fr/viewtopic.php?f=7&t=5129) but many others are also included.

the main questions are:
1. is the view now that CABAC is available, and should be chosen, for TV? if so, to what level does this go up to? from my testing, of 1 scene, this never produces smooth action...
2. if the above is true, how about "trellis"? marginal differences between the 0, 1 and 2 levels, but OMG! its so much better than CAVCL!
3. suggested "advanced option strings"? i have a base, as you will see below...

i have been testing the following "advanced option strings", on the chase scene near the start of Casino Royale (00:11:00 - 00:15:00) to get the panoramic shots and fast pace, with some close detail too.
with constant quality of circa 59%, picture output size as near as 1280x720 as possible retaining aspect ratio's, dolby pro logic 2 at 160kbps, frame rates "same as source", i have then tried these options:
ref=3:bframes=6:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:cabac=0 (very smooth. lacking, well atleast less, detail)
ref=3:bframes=6:subq=6:me=umh:mixed-refs=1:keyint=240:keyint-min=24:cabac=0 (same as above)
ref=3:bframes=6:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=0 (choppy. running through jungle and explosion/flames are particularly bad)
ref=3:bframes=6:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=1 (choppy. running through jungle and explosion/flames are particularly bad)
ref=3:bframes=6:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=2 (choppy. choppy. running through jungle and explosion/flames are particularly bad)
ref=3:bframes=3:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=2 (choppy. choppy. choppy! running through jungle and explosion/flames are particularly bad)
ref=3:bframes=3:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=2:b-adapt=2 (slightly choppy, mainly the above scenes. the panoramic shots on top of crane less jerky)
ref=3:bframes=3:subq=6:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=0:b-adapt=2 (slight improvement from above)
ref=3:bframes=3:subq=7:me=umh:mixed-refs=1:keyint=120:keyint-min=24:trellis=0:b-adapt=2 (follows on from above!)

hoping someone, anyone, will add to this to help out...

regards

jingo_man

[dynaflash]

while 62% is considered transparent for dvd sources now (back in those days it was about 70%, but the x264 crew added psy-rd and some other cool stuff .. well anyway ...), however, for hd sources I seriously suggest backing down to somewhere to the tune of 55% to 57% as you are feeding HB a much higher rez source than dvd.

Oh,and if you aren't already using the dev snapshot ... go get it.

[jingo_man_dan]

thanks dyna

i have amended the CQ down to 57%, and will go from there, tweaking all the other options as i go aswell...

think it was the one thing i forgot to detail was the version of HB i was using! i compiled from the svn last week, so the version i have is "svn2727 i386 (2009072401)".

i also see from the threads that there are particular problems with specific films. is casino royale one of them? are there any recommendations for films to use for a really good "test" scene of a few mins? or have i chosen well?

also, are there recommended mac os x media players to record bitrates whilst a film is playing? i would imagine VLC would be the choice. any suggestions of how to continually record bitrates, so it can be graphed, etc?

cheers again

jingo_man

[jno]

I have also been playing around trying to find the best ATV settings for an HD source. The maximum I've been able to pump CQ up to without noticeable studdering is 59% (i'll post my settings when I get home, mostly based of the older ATV advanced settings threads). This is without CABAC enabled, I am going to try and do some new tests tonight with CABAC enabled to try and find the maximum I can push the ATV.

As for watching the playback, I usually use VLC for playback because I find it actually looks a lot better then quicktime playing back my files. This is probably because VLC is newly updated, hopefully quicktime 10 in snow leopard does a better job of playback and handles more containers. if you hit command + i when the movie is playing you can see the live bitrate of the file(make sure the movie is selected in your playlist... I find it annoying it doesn't auto select what is playing...), I will use quicktime to find the average bitrate of a HD encode to make sure its not too high (anything over 5500 kbps I usually see studdering).

[jingo_man_dan]

having repeated numerous encodes from the same scene of casino royale, i have finally stumped for the following advanced options, with 55% CQ, AAC (Core Audio) to DPL2 @ 160kbps:

ref=4:bframes=3:subq=9:me=umh:mixed-refs=1:keyint=120:keyint-min=24:b-adapt=2:direct=auto:8x8dct=1:analyse=all

so far, all encodes look great and without any issue of lagging, locking, freezing, etc. ffd, rewind and chapter markers all execute without issue too.

i would imagine some films can be further tweaked but i am more than happy with the results for this to become my own "preset", to allow a run-encode-once, sit-back-and-play scenario

it would still be interesting to know the bit-rates of any encode, sampling throughout its playback (particularly natively on the appletv so that it can be graphed and due to its own use of GPU processing compared to a Mac desktop/laptop which will likely process differently...) but i dont currently know if this is even possible, let alone how to actually do it! i dont imagine standard system counters are available for quicktime bit-rates for mac os x... let alone appletv's tailored version of the O/S.

to jns, stick with CABAC. it simply makes the results awesome by comparison!! and i found that i had issues with most of my test advanced settings strings even choosing 57% with CABAC enabled, though i was using a particularly action-packed, explosions, panoramic's based footage, to give my tests what i thought would be "top-end" requirements... with lesser films (and/or 720p or 1080i broadcast-quality footage, higher settings could be attained. unfortunately with such a wide spec of what is "HD footage", this could very likely be an endless pursuit, scientifically! haha). with dyna's input, switching to 55% produced much better results with negligible deterioration.

now, that full casino royale encode that's just finished isnt gonna watch itself....

jingo_man

[jno]

@jingo_man_dan great work there! I played around with those setting tonight with the first minute and a half of Quantum of Solace which is a tricky encode. I still can't get it not to studder at the scene where it switches from the opening to the lake scene, but its looking a lot better. Now I need to decide if subq=9 is worth it (aprox 10% longer encode time over subq=7), I can sorta tell the difference on my computer but not on the ATV.... also figuring out if I need trellis=1 or not.

my settings are: ref=4:bframes=3:subq=7:me=umh:mixed-refs=1:keyint=120:keyint-min=24:b-adapt=2:direct=auto:8x8dct=1:analyse=all:trellis=1 at 55% CQ

CABAC is simply awesome, cuts bitrates by a large amount. really impressed.

ohh and if you are putting in the time to encode these beautiful HD encodes you need to include some AC3 surround sound... just saying.

[jingo_man_dan]

@jno, thanks for the "big up"... i know the scene you are on about! i will be digging up QoS some point soon to redo this film with these settings, so will have to see how i get on...

yeah, my last decision was between subq=9 or 7... but i thought that even though "restricting" the encode to a max of 3 consecutive b-frames (bframes=3) that to have Psy-RDO being applied on every frame was more than worth the additional encode time (i normally leave mine running overnight or whilst at work, so not too bothered by this - and would rather get a good encode 1st time around, as otherwise it will be significantly more than 10% timing to encode for a 2nd time...!) though i must confess i dont know much about these latest x264 features as this is the 1st time i have revisited my encoding profiles in couple years (see opening post!)

trellis'ing... again, i dont know much, but from the encodes i did, it really made my poor, little appletv struggle on fast-paced and panoramic scenes. slower/normal speed motions were delicious though... but for me, its pushing it too high for a "high end, but fits all" type of preset. additionally, as you can see from my advanced string, i didnt apply the additional "psy-rd=x,y" value either, so not sure if this was being applied as intended, or if i was getting the best out of the feature... maybe dynaflash, or others, could help explain this for us/me..?

as for the sound - i know its a reduction... would you recommend AC3 passthrough then? i dont have an AV Reciever at this moment, so not able to take advantage of the sound at this point anyway, so thats probably why i havent done much around sound options... unfortunately, i need a new bathroom suite first, which is stopping me purchasing that Onkyo i have my eye on... hehe

jingo_man

[jno]

I guess I am going to go with subq=9, with how limited the ATV is with HD sources you really need to squeeze every ounce of quality out of your encodes to be happy with it. I would be much happier if the ATV could handle a CQ of 59 with CABAC....

I guess I'll drop trellis I did notice my ATV struggle a bit more with trellis=2 compared with trellis=0, but trellis=1 didn't really produce a noticeable difference. As with you my knowledge of advanced x264 options is limited but I'm trying to learn, and would welcome some help. I believe psy-rd=1,0 is the default, don't really know much about it or if increasing it to psy-rd=1,0.1or0.2 would help much.

I would totally recommend AC3 passthrough if your source has AC3, somewhere in the future you may invest in the awesomeness of surround sound, and you are going to notice the difference between stereo and 5.1. A lot of the new HD scene releases on the internets come with DTS audio, for that you are going to have to convert it to AC3 which on windows is easy with: MKV Audio Converter. if you are on a mac here is a guide how to get it working: http://www.mymediadelivered.com/2008/07 ... -apple-tv/

But if any of the handbrake veterans can chime in and see if we are missing anything or doing something totally wrong with our HD (720p) settings for the ATV, let us know.

[dynaflash]

for my part it's generally the same as the string I use from the existing advanced appletv thread, however I do not specify keyint or keyint-min as hb does a pretty decent job of it on it's own. Other than that I can say those settings should squeeze about all you are likely to get from an AppleTV currently. Though 3.1 Hi Profile is nothing to sneeze at really. Weighted bframes would be nice, but when I tested with it I did get some stuttering on occasion, sadly I found out after the 0.9.3 preset was already released, though admittedly have not retried it since the last two atv software updates. Hmmm... any brave souls ?

[jno]

Weighted b-frames is a no-go. I've tried it and get heavy stuttering.

I must say these settings are about as good as you are going to get, before I was going with the method of throwing as high of a bitrate at the ATV, with minimal advanced settings. These looked good but for almost all movies I ran up to the 4gb cap (want to play them on my 360 as well, ATV at home 360 at school). Now I'm averaging about 2.5GB per 120min HD movie and 1.5GB for DVD movie, which is really saving on hard drive space.

[dynaflash]

I was afraid of that. Oh, well. For what it is you can still get some pretty decent looking video on the AppleTV with those options imo.

[jno]

Not to get too off topic... but whats so good about weighted b-frames? for those trying to learn about the advanced x264 settings.

[dynaflash]

Not to get too off topic... but whats so good about weighted b-frames? for those trying to learn about the advanced x264 settings.

http://trac.handbrake.fr/wiki/x264Options#options

[jno]

Update, So I've run some more encodes and it looks like the ATV is starting to struggle with full framed HD encodes (ie 1280X720) using your original settings. It also struggling even more with AC3 surround sound at your original settings (which i guess is understandable seeing as surround sound take more CPU to process). I have to drop the CQ down to 49-51% to get a smooth playback with surround sound and CABAC, at that level I can noticeably see some artifacting and reduced quality. Looks like I am going to have to turn CABAC off to keep surround sound and bump the CQ up to 59%. This looks a lot better but with about 30-40% more bitrate per movie. sigh.... no CABAC awesomeness for my ATV HD encodes.

[peterjcat]

Is it not the case that using CABAC at a given CRF has no bearing on the quality of the resulting video, only on its bitrate/filesize? If it is the case, then if you're not too worried about filesize then wouldn't CRF at 57%-59% without CABAC be better than CRF at 55% with CABAC? I settled on that a while ago anyway, since the Apple TV seemed to stutter at least once on most of my CABAC encodes at 57% or so.

Oh, and forget about weighted B-frames, although for me the problems only turned up during the opening titles or any other time there was fading text, for some reason—and I haven't tried since Apple TV 2.2 either.

[dynaflash]

Is it not the case that using CABAC at a given CRF has no bearing on the quality of the resulting video, only on its bitrate/filesize?

Correct. A cabac encode and a cavlc encode will look the same if everything else is equal. Typically the cabac encode will be around 20% smaller or so at the same visual quality.