ATV2: Reducing macroblocking in dark scenes?

[Scott R]

I mentioned this in another thread, but thought it might warrant a thread dedicated to it. I recently experimented with some of the fast, faster, veryfast, etc. settings combined with the default ATV2 preset, and also experimented with changing the CQ RF level. My test Blu-ray was the first Iron Man movie, seconds portion 1980 - 2280 (00:33:00 - 00:38:00 - 5 mins total). These are cave scenes towards the early part of the movie, so they're very dark, and are "good" for spotting macroblocking issues in large swaths of black areas. At the 2 min 07 sec portion of this test snippet (00:35:07 of the movie), the lights dim to black with a laptop screen centered on the screen. Looking to the left of the laptop screen, you can see noticeable macroblock "squares" using the RF 20 setting, and using any of the fast/faster/veryfast settings at RF 20 make it even worse.

I found that I needed to get down to RF 16 (normal speed settings) for the blocks to get to an acceptable level. RF 14 was even better when viewed up close to the screen, but I the significantly larger filesize made it not worth the tradeoff for me. Unfortunately, after ripping a few movies at RF 16, I discovered that my filesize estimations were off and, in some cases, the filesizes were a good deal larger than I thought they'd be. For example, the movie Source Code ended up being about 6.2GB, and that's only a 93 minute movie. Had it been 2 hours or longer, it would have exceeded the ATV2's 8GB buffer (though I'm not sure if that would have broken compatibility).

Given that using RF 20 combined with the veryfast setting produced a slightly smaller, but more noticeably lower PQ, video than the standard RF 20 settings, I decided to experiment with the slow/slower/veryslow settings today. I started out with the ATV2 preset (RF 20), and then added the veryslow advanced settings, with the intent of then repeating the experiment with an RF 18 and RF 16 value. But the first test failed because it did not play smoothly on the ATV2.

So am I stuck with having to use an RF value of no less than 18 in order to safely ensure that the filesize will fit within the ATV2's buffer, even though it might (as in the case of Iron Man) have noticeable macroblocking in dark scenes? Or are there other settings that I can use which will reduce this macroblocking while still keeping filesizes manageable?

[Scott R]

Update: I went back and reviewed the veryslow settings and it looks like my problem was that the ref and bframes settings shouldn't be overridden for ATV2 compatibility. So I just tried another encode, this time using the ATV2 preset, RF 18 override, and the veryslow advanced settings (minus the ref and brames overrides), and that seemed to play OK. The filesize was slightly smaller than the standard RF 18 encode, and the PQ seemed to be somewhat better, but the encode time was significantly longer (around 13fps vs 32fps).

Not sure where I'll end up on this.

[Deleted User 11865]

It should be OK to use more refs than the default, but don't forget that the AppleTV 2 only officially supports levels up to 3.1.

Have a look at H.264 Levels

[Scott R]

Been busy with work, power outages, and other things, so I'm just checking back on this...

It should be OK to use more refs than the default, but don't forget that the AppleTV 2 only officially supports levels up to 3.1.

Have a look at H.264 Levels

Will increasing refs directly correlate to an improvement in the macroblocking I've described?

[Deleted User 11865]

No, it'll just increase compression efficiency.

[Scott R]

Thanks. Do you have any thoughts on which settings, if any, would have the most effect on reducing this macroblocking? Or is decreasing the RF value pretty much it?

[Deleted User 11865]

JohnAStebbins has some suggestions which may help regarding this particular issue. You will have to use forum search, as I don't remember which topic(s) may feature his suggestions.