H.264 vs H.265 preset expectations?

[DataMeister]

I took a 20 minute camcorder video that was about 4 GB and ran it through two different Handbrake Presets for comparison. I'm curious what I should be expecting here.

Presets:

H.264 MKV 1080p30 - resulted in a 669,565 KB file size.
H.256 NVENC 1080p30 - resulted in a 1,724,288 KB file size.

I was expecting the H.265 file size to be smaller or about equal size with better quality. What should I be expecting? Are the presets supposed to be relatively close in quality? Is the H.265 NVENC encoder just that much worse that it needs more than double the file sizes of the H.264 software encoder?

[mduell]

You're keeping nothing constant, so I'm not sure what you could expect. The quality scales aren't the same for the two encoders, and there's no simple conversion between them. Depending on your frame size and quality target, H.265 may not may not have an advantage over H.264. With a hardware encoder, for any codec, quality for size takes a bit of a loss.

[DataMeister]

You're keeping nothing constant, so I'm not sure what you could expect. The quality scales aren't the same for the two encoders, and there's no simple conversion between them. Depending on your frame size and quality target, H.265 may not may not have an advantage over H.264. With a hardware encoder, for any codec, quality for size takes a bit of a loss.

Well first the source is constant and the output resolution is constant.

But, I expected the quality scales to be different just from looking at how H.264 is set to CQ: 22 and the H.265 NVENC is set to CQ: 27. I figured a bit of a loss from hardware rendering might would be 10% bigger than the software equivalent but not over double the size of a previous generation codec. That made me wonder if something was screwed up.

If there is not a simple conversion, the question is how were those presets decided upon to be included in Handbrake?

[Deleted User 11865]

They were probably calibrated on sources that were quite different to the one you used here You may indeed have to adjust the CQ to your own liking.

And it's quite possible for a software H.264 encoder to beat a hardware H.265 encoder (not on all sources and not at all bitrates, but it's probably more frequent than you would expect).

Especially when targeting "high quality" and/or "perceptual transparency", even a software H.265 encoder may not beat an otherwise equivalent H.264 software encoder by more than 10% or thereabouts (and there may be cases where the H.264 encoder beats the H.265 encoder, although if both software encoders are decent, this will be less frequent).

[DataMeister]

Just as a follow up, I ran the H.265 MKV 1080p30 software encoder preset which, like the H.264 preset also uses CQ: 22, and the file size ended up about half that of the H.264.

H.265 MKV 1080p30 - 321,880 KB.

[mduell]

The quality scale isn't the same across the two encoders.