Better settings don't always equal better encodes

Post your testing results with HandBrake.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Better settings don't always equal better encodes

Post by Arlki »

PLEASE NOTE! I'm testing against constant quality mode (RF=19) as it seems to be recommended over guesstimating a file size / bitrate.
I suspect the quality score is not exact and that this might have some affect on the results when used in different encodes of the same source material, but I do not believe that it will invalidate the final conclusion.
(At least a quick visual inspection is enough for me to say "that looks fine" with almost everything encoded at rf=19.)

TEST PC: Athlon x2 3800+ running with 3GB DDR400 CAS3 Command Rate T2.

TEST PROCEDURE: In order to reduce the amount of time required I repeatedly re-encoded the same anime opening (a 1280*720 version of CoalGirl's Katanagatari opening) whilst each time changing one setting. The non-variable settings are as follows :

Constant Quality RF:19 / single pass; mixed-refs=1:no-fast-pskip=1:no-dct-decimate=1:8x8dct=0.

The fast-pskip and no-dct-decimate were left enabled as they can have an affect on quality that I wouldn't be taking into account. (I was only interested in speed and size.)

Cabac was always enabled and 8x8 decimation was always disabled.
(Both of these decisions were due to the results of a previous benchmark fest (now lost) - cabac gave large size decreases, and against my expectations, 8x8 dec gave size increases even though I verified the correct setting in the handbrake log files. I am not sure why this was the case - perhaps it was because of the source material.)

TEST RESULTS : In order to make these easier to read, I have normalized all results against those for the following encode :

bframes=8:subq=0:mixed-refs=1:trellis=0:b-pyramid=1:no-fast-pskip=1:no-dct-decimate=1:ref=1:8x8dct=0:me=dia.
Average fps = 23.28133; file size = 39709431bytes.

This means that an encode with FPS = 2 and SIZE REDUCTION = 20 took twice as long, but only reduces the final file size by 20%. The efficiency score is the percentage file reduction divided by the fps score - the negative scores are where a file ended up being larger than my standard.

The settings column shows :
Reference Frame - B Frames - Weighted P Frames - Adaptive B Frames - Motion estimation - Subpixel Mode - Trellis

NOTE! :
I will be using these results to decide on my own (animation) encoding settings.
For live action stuff / other machines etc, there may be different best settings.
I'm just posting these as it took a while for me to do, and as a warning against going straight for the "best" settings.

Settings ==== fps ==== size ==== Efficiency
==================================================================
1 08 0 Fast Diamond 00 0 ==== 1.01 ==== -0.20 ==== -0.20
1 08 1 Fast Diamond 00 0 ==== 1.00 ==== 0.00 ==== 0.00
->
Enabling weighted P Frames increased the fps and reduced file size, perhaps due to the number of fades.
I expect that this only holds true when encoding to a quality limit.
==================================================================
1 08 0 Fast Diamond 00 0 ==== 1.01 ==== -0.20 ==== -0.20
1 08 0 Optimal Diamond 00 0 ==== 1.98 ==== 2.30 ==== 1.16
------------------------------------------------------------------
3 16 1 Fast Diamond 00 0 ==== 1.05 ==== 4.60 ==== 4.38
3 16 1 Optimal Diamond 00 0 ==== 4.51 ==== 4.90 ==== 1.09
->
Setting adaptive b frames to optimal does decrease file size but has a far larger impact on fps.
==================================================================
1 16 1 Fast Diamond 00 0 ==== 1.48 ==== -0.80 ==== -0.54
1 16 1 Fast Diamond 00 1 ==== 1.25 ==== 1.90 ==== 1.52
1 16 1 Fast Diamond 00 2 ==== 1.25 ==== 1.90 ==== 1.52
------------------------------------------------------------------
3 16 1 Fast Diamond 00 0 ==== 1.05 ==== 4.60 ==== 4.38
3 16 1 Fast Diamond 00 1 ==== 1.79 ==== 6.40 ==== 3.58
------------------------------------------------------------------
3 16 1 Fast Diamond 02 0 ==== 1.52 ==== 17.20 ==== 11.32
3 16 1 Fast Diamond 02 1 ==== 2.20 ==== 18.70 ==== 8.50
------------------------------------------------------------------
3 16 1 Fast Diamond 03 0 ==== 1.76 ==== 19.60 ==== 11.14
3 16 1 Fast Diamond 03 1 ==== 2.44 ==== 20.30 ==== 8.32
------------------------------------------------------------------
3 16 1 Fast Diamond 04 0 ==== 2.71 ==== 20.90 ==== 7.71
3 16 1 Fast Diamond 04 1 ==== 2.85 ==== 20.50 ==== 7.19
------------------------------------------------------------------
3 16 1 Fast Diamond 05 0 ==== 3.17 ==== 22.10 ==== 6.97
3 16 1 Fast Diamond 05 1 ==== 3.39 ==== 21.70 ==== 6.40
->
In all but the first encode, enabling trellis decreased the encode fps.
In all but the last 2, it also gave a file size decrease; but in every case the encoding efficiency was hurt.
==================================================================
1 16 1 Fast Diamond 00 0 ==== 1.48 ==== -0.80 ==== -0.54
2 16 1 Fast Diamond 00 0 ==== 1.05 ==== 3.60 ==== 3.43
3 16 1 Fast Diamond 00 0 ==== 1.05 ==== 4.60 ==== 4.38
------------------------------------------------------------------
1 16 1 Fast Diamond 01 0 ==== 1.05 ==== 8.50 ==== 8.10
2 16 1 Fast Diamond 01 0 ==== 1.20 ==== 10.80 ==== 9.00
3 16 1 Fast Diamond 01 0 ==== 1.18 ==== 11.60 ==== 9.83
------------------------------------------------------------------
1 16 1 Fast Diamond 10 2 ==== 8.43 ==== 20.90 ==== 2.48
2 16 1 Fast Diamond 10 2 ==== 8.54 ==== 23.70 ==== 2.78
3 16 1 Fast Diamond 10 2 ==== 8.74 ==== 24.50 ==== 2.80
->
Increasing reference frames decreases both fps and file size.
(I enabled trellis=always on for the sub=10 encodes as handbrake said it should always be on.)
==================================================================
3 16 1 Fast Diamond 00 0 ==== 1.05 ==== 4.60 ==== 4.38
3 16 1 Fast Hexagonal 00 0 ==== 1.08 ==== 5.10 ==== 4.72
3 16 1 Fast UMH 00 0 ==== 1.48 ==== 6.60 ==== 4.46
------------------------------------------------------------------
3 16 1 Fast Diamond 10 2 ==== 8.74 ==== 24.50 ==== 2.80
3 16 1 Fast UMH 10 2 ==== 9.20 ==== 25.80 ==== 2.80
->
Increasing the motion estimation method reduced the fps but did not always reduce file size.
==================================================================
1 16 1 Fast Diamond 01 0 ==== 1.05 ==== 8.50 ==== 8.10
1 16 1 Fast Diamond 02 0 ==== 1.33 ==== 13.60 ==== 10.23
1 16 1 Fast Diamond 03 0 ==== 1.53 ==== 15.60 ==== 10.20
1 16 1 Fast Diamond 04 0 ==== 1.79 ==== 15.70 ==== 8.77
1 16 1 Fast Diamond 05 0 ==== 2.06 ==== 16.70 ==== 8.11
1 16 1 Fast Diamond 06 0 ==== 2.30 ==== 12.50 ==== 5.43
1 16 1 Fast Diamond 07 0 ==== 2.43 ==== 13.50 ==== 5.56
1 16 1 Fast Diamond 08 0 ==== 2.99 ==== 12.80 ==== 4.28
1 16 1 Fast Diamond 09 0 ==== 3.18 ==== 12.60 ==== 3.96
1 16 1 Fast Diamond 10 2 ==== 8.43 ==== 20.90 ==== 2.48
2 16 1 Fast Diamond 00 0 ==== 1.05 ==== 3.60 ==== 3.43
2 16 1 Fast Diamond 01 0 ==== 1.20 ==== 10.80 ==== 9.00
2 16 1 Fast Diamond 02 0 ==== 1.49 ==== 16.50 ==== 11.07
2 16 1 Fast Diamond 03 0 ==== 1.72 ==== 18.80 ==== 10.93
2 16 1 Fast Diamond 06 0 ==== 2.79 ==== 15.80 ==== 5.66
2 16 1 Fast Diamond 08 0 ==== 3.45 ==== 16.00 ==== 4.64
2 16 1 Fast Diamond 10 2 ==== 8.54 ==== 23.70 ==== 2.78
3 16 1 Fast Diamond 00 0 ==== 1.05 ==== 4.60 ==== 4.38
3 16 1 Fast Diamond 01 0 ==== 1.18 ==== 11.60 ==== 9.83
3 16 1 Fast Diamond 02 0 ==== 1.52 ==== 17.20 ==== 11.32
3 16 1 Fast Diamond 03 0 ==== 1.76 ==== 19.60 ==== 11.14
3 16 1 Fast Diamond 04 0 ==== 2.71 ==== 20.90 ==== 7.71
3 16 1 Fast Diamond 05 0 ==== 3.17 ==== 22.10 ==== 6.97
3 16 1 Fast Diamond 06 0 ==== 2.93 ==== 16.70 ==== 5.70
3 16 1 Fast Diamond 08 0 ==== 3.65 ==== 17.00 ==== 4.66
3 16 1 Fast Diamond 10 2 ==== 8.74 ==== 24.50 ==== 2.80
3 16 1 Fast Hexagonal 00 0 ==== 1.08 ==== 5.10 ==== 4.72
3 16 1 Fast Hexagonal 01 0 ==== 1.20 ==== 12.00 ==== 10.00
3 16 1 Fast Hexagonal 02 0 ==== 1.56 ==== 17.80 ==== 11.41
3 16 1 Fast Hexagonal 03 0 ==== 1.82 ==== 20.10 ==== 11.04
3 16 1 Fast Hexagonal 04 0 ==== 2.71 ==== 21.10 ==== 7.79
3 16 1 Fast Hexagonal 05 0 ==== 3.25 ==== 22.30 ==== 6.86
3 16 1 Fast Hexagonal 06 0 ==== 2.95 ==== 17.20 ==== 5.83
3 16 1 Fast Hexagonal 08 0 ==== 3.81 ==== 17.50 ==== 4.59
3 16 1 Fast UMH 00 0 ==== 1.48 ==== 6.60 ==== 4.46
3 16 1 Fast UMH 10 2 ==== 9.20 ==== 25.80 ==== 2.80
==================================================================
ANOMALIES:
3 16 1 Fast Hexagonal 10 0 ==== 4.02 ==== 26.70 ==== 6.64
->
I set trellis to zero by mistake, which greatly improved the fps.
Unfortunately I don't know if the file was viewable, because I deleted it before realizing my mistake.
==================================================================
CONCLUSIONS:
Simply increasing encoding settings is almost never the most efficient way to do things.

For example:
3-16-1-Fast-UMH-10-2 reduces file size by 25.8% but 9.2x longer than the normalized encode.
3-16-1-Fast-Hexagonal-05-0 reduces file size by 22.3% but only takes 3.25x longer than the normalized encode.
(This gives an efficiency of 2.80 vs 6.86.)

Sometimes, it even leads to an increase in file size or encoding time with no benefit.
3-16-1-Fast-Diamond-02-0 => 17.2% in 1.52x normalized time.
3-16-1-Fast-Hexagonal-06-0 (My old favourite!) => 17.2% in 2.95x normalize time! :oops:

If speed is an issue as well as file size, then 3-16-1-Fast-Hexagonal-02-0 gives almost 18% for only 1.56x the normalized encode. If speed is a major issue, then 3-16-1-Fast-Hexagonal-01-0 gives 12% reduction for only 1.2x time.
===
Unfortunately I'm not sure how to upload a graph, but mapping size reduction against fps clearly shows that some encode settings don't seem to be worthwhile. (I expect this will hold true for live action stuff as well, but I'm not sure whether the same encodes would appear to be best.)
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: Better settings don't always equal better encodes

Post by jbrjake »

That's funny, I don't see a single mention of "SSIM" in your entire multi-page post ostensibly about "better encodes" and whether a given setting is "worthwhile."
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

@ jbrjake
Though I've seen ssim and snr values in the log files, I've seen no mention of them in the handbrake gui.

The problem for me then was that I had no way to aim for a certain ssim value, nor in fact the knowledge to understand how significant the differences in attained ssim values might be. Given that, I decided to follow the handbrake recommendation and use the constant quality RF mode; and from there investigate firstly how long each encode would take, and secondly how large the resulting file would be. (Primarily because my pc is too slow for me to simply put all of the settings up to their highest level.) My intention in posting the results of my work was to aid those other people like myself who might wish that they could somehow encode things faster than they are at the moment without sacrificing quality.

Of course, given my lack of knowledge reference h264 quality measurements - I know only that snr values don't tend to show the whole picture - then I had to rely on the assumption that the handbrake engine would use sufficient bitrate to ensure that the quality of each rf=19 encode would be approximately equal regardless of what settings I used. If that is not the case then perhaps someone else with more knowledge of those things, or perhaps even the handbrake developers, would like to map out how rf=(whatever value) corresponds to a files real quality at each combination of settings; but until someone does that then people like myself will have to trust the constant quality mode to do as it says.
User avatar
Rodeo
HandBrake Team
Posts: 12757
Joined: Tue Mar 03, 2009 8:55 pm

Re: Better settings don't always equal better encodes

Post by Rodeo »

Arlki wrote:then I had to rely on the assumption that the handbrake engine would use sufficient bitrate to ensure that the quality of each rf=19 encode would be approximately equal regardless of what settings I used.
This is not true and makes your whole analysis pointless (note: I did make the same assumption, a long time ago).

One good example: in my experience, 8x8dct almost always increases the file size (especially for grainy sources, but also for all sources in general, even if to a lesser extent), but always increases quality, often significantly.
Arlki wrote:If that is not the case then perhaps someone else with more knowledge of those things, or perhaps even the handbrake developers, would like to map out how rf=(whatever value) corresponds to a files real quality at each combination of settings; but until someone does that then people like myself will have to trust the constant quality mode to do as it says.
It's not easy and I doubt there can be an exact mapping. But the point it, at the same rate factor (RF), settings A and B won't necessarily result in the same quality, so using a constant RF cannot be used to compare compression efficiency (or compression efficiency per speed, or whatever you feel like).
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

Rodeo wrote:
Arlki wrote:
then I had to rely on the assumption that the handbrake engine would use sufficient bitrate to ensure that the quality of each rf=19 encode would be approximately equal regardless of what settings I used.
This is not true and makes your whole analysis pointless (note: I did make the same assumption, a long time ago).
Please make note of my use of the word "approximately".

I was already well aware that there were some differences in achieved quality by looking at the log files; the point is that unless a handbrake user can say not only that "setting a rf19 quality always exceeds setting b rf19", but also that "setting a rf19 quality exceeds setting b rf18.5 quality" then comparing different settings at rf19 is still valid and far from pointless. Given that lots of people state "rf(xx) is overkill" without qualifying the settings then I feel justified in stating that if you want to aim for a certain rf value, and if your pc isn't quick enough to just max all of the settings, then it doesn't hurt to compare different ways of encoding to that quality setting. Of course, if people then say "RF(xx) with (8x8 / no-fast-pskip / whatever) is all you need" then that's a different matter. :D
Rodeo wrote:
One good example: in my experience, 8x8dct almost always increases the file size (especially for grainy sources, but also for all sources in general, even if to a lesser extent), but always increases quality, often significantly.
Then are there any other settings that an encoder should use whose benefit to the quality of a video will not be recognised by handbrake when it tries to hit a constant quality rf value?

You might have noticed that in the same way that you always include 8x8, I included no-fast-pskip and no-dct-decimate precisely because I have heard that they can help in some cases even though they can also hurt the encode rate. Does ME=Dia, SUB=0 plus 8x8 beat ME=UMH, SUB=0? I (and probably plenty of other people) have only so much cpu time to spare but would like to get as much quality out of that as possible. If you say that my whole analysis was pointless, I would be grateful if you could point me to something more useful.

My previous favorite (3-16-1-HEX-6-0) was selected after reading the mencoder guide and other resources; but in all of those the cpu consumption figures were either vague (increases by 30% reference to some unknown starting point) or depended too much upon bitrate / b-frames etc. My analysis was a way for me to discover, for my usual source material, if I could save some cpu cycles and some space at the same time.
Rodeo wrote:
Arlki wrote:
If that is not the case then perhaps someone else with more knowledge of those things, or perhaps even the handbrake developers, would like to map out how rf=(whatever value) corresponds to a files real quality at each combination of settings; but until someone does that then people like myself will have to trust the constant quality mode to do as it says.
It's not easy and I doubt there can be an exact mapping. But the point it, at the same rate factor (RF), settings A and B won't necessarily result in the same quality, so using a constant RF cannot be used to compare compression efficiency (or compression efficiency per speed, or whatever you feel like).
Please again note "approximately". Unfortunately I deleted the log files after reading the fps from then, so I can't now compare ssim values; but even if I could :
(a) I have no way of aiming for a set ssim value without doing multiple encodes. (Does this hit it, is this too much etc)
(b) I am going to guess that most likely there would be never more than 0.5 rf points difference between two encodes. (IE setting b rf 19 ssim is always better than setting a rf 19.75 and worse than setting a rf 18.25.)

Anyway, if you can point me to a more detailed setting-by-setting comparison that works on the constant-quality setting rather than some guesstimated bitrate, then I would be very grateful. If you cannot, then perhaps what I did wasn't pointless? :D
User avatar
JohnAStebbins
HandBrake Team
Posts: 5588
Joined: Sat Feb 09, 2008 7:21 pm

Re: Better settings don't always equal better encodes

Post by JohnAStebbins »

Anyway, if you can point me to a more detailed setting-by-setting comparison that works on the constant-quality setting rather than some guesstimated bitrate, then I would be very grateful
You're not going to find such a comparison, because the video source also plays an important part in how effective some settings are. Some settings will work better on some video sources than on other sources. If you want to do some kind of statistically meaningful analysis, you would have to take a large number of video sources, encode them all with every permutation of settings, and plot the average ssim for each permutation.
(b) I am going to guess that most likely there would be never more than 0.5 rf points difference between two encodes. (IE setting b rf 19 ssim is always better than setting a rf 19.75 and worse than setting a rf 18.25.)
Your guess is quite incorrect. Compare the output of x264's ultra fast preset to the output of high profile. The quality difference it quite large for the same RF.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

John A Stebbins wrote:
Anyway, if you can point me to a more detailed setting-by-setting comparison that works on the constant-quality setting rather than some guesstimated bitrate, then I would be very grateful
You're not going to find such a comparison, because the video source also plays an important part in how effective some settings are. Some settings will work better on some video sources than on other sources. If you want to do some kind of statistically meaningful analysis, you would have to take a large number of video sources, encode them all with every permutation of settings, and plot the average ssim for each permutation.
(b) I am going to guess that most likely there would be never more than 0.5 rf points difference between two encodes. (IE setting b rf 19 ssim is always better than setting a rf 19.75 and worse than setting a rf 18.25.)
Your guess is quite incorrect. Compare the output of x264's ultra fast preset to the output of high profile. The quality difference it quite large for the same RF.
I couldn't find the "ultra fast" setting that you mentioned, even when I reinstalled v3501 (v3505 wouldn't load videos at all); but I ran various encode settings (including normal and high profile) and did indeed see that I was wrong. :oops: m(_ _)m

People seem to say that SSIM is more meaningful that PSNR, though not entirely accurate, so I'll list the attained values. Please let me know if there is a better (non-visual) way of measuring quality. m(_ _)m

(These results are not based on the file in my previous analysis.)
(SETTING A = bframes=16:subq=2:mixed-refs=1:trellis=0:b-pyramid=1:no-fast-pskip=1:no-dct-decimate=1:ref=3:8x8dct=0:me=hex)

RF=20 : SSIM Mean Values :
Normal = 0.9838319 (17.913db) / ?? FPS
High = 0.9847115 (18.156db) / 19.19 FPS
SETTING A = 0.9838170 (17.909db) / 46.00 FPS
SETTING A + 8x8 = 0.9843032 (18.042db) / 44.65 FPS
SETTING A + Sub:6 = 0.9843501 (18.055db) / 23.66 FPS
SETTING A + 8x8 + Sub:6 = 0.9849106 (18.213db) / 22.60 FPS

(To confirm that I was wrong, I checked an encode with the worst performing settings at RF19.25 instead of RF20, and it still couldn't match the best encode at RF20.)

As an experiment, I also looked back at some of my old log files, and found that for a range of files under exactly the same encode settings I attained SSIM values ranging from 19.4 to 21.1db.
I don't have tons of time on my hands, but I'm considering re-running the benchmark fest with a target SSIM value in mind; though it would again be limited to a single source file. (IE this time I would change settings, benchmark the encode then alter the RF value in attempt to reach some required SSIM value. Once the target value had been obtained I would then record the rf value and the fps and enter those against the settings used - from the results of these I would then work out which encode settings to use.)

If you have an opinion on this, please could you suggest an SSIM value that I should aim for as a minimum? At what level would most people find it very hard to see a difference without examining the encodes purely for comparison purposes? (I know that this is highly subjective, but I would be encoding animated material where sharp / non-artifacted lines and a lack of non-obvious banding would be preferred.)
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

A quick preview of the new benchmark results.
(This is in preparation for a SSIM targeted benchmark; tests are on the same athlon x2 3800+. Encode source is a 1280*720 opening with lots of (fairly repetetive) movement.)

RF=19;
=====================
bframes=16:8x8dct=0:subq=(xxx):trellis=0
(xxx) === (SSIM) === (FPS)
-----------------------------
0 === 21.455 === 21.6
1 === 21.799 === 19.8
2 === 22.397 === 16.6
3 === 22.792 === 14.3
4 === 22.830 === 11.9
5 === 22.826 === 10.2
6 === 22.761 === 9.4
7 === 22.741 === 8.8
8 === 22.842 === 7.8
9 === 22.876 === 7.4
=====================
bframes=16:subq=1:trellis=0
-----------------------------
0 === 21.361 === 21.0
1 === 21.696 === 19.0
2 === 22.349 === 15.6
3 === 22.742 === 13.2
4 === 22.777 === 10.9
5 === 22.771 === 9.1

Notice how the fps drops continually, but (at least for this source) there appears to be a local maximum ssim at subpix = 4. Have set up tests with 8x8 enabled, no-fast-pskip and hexagonal motion estimation. (Will add no-dct later.) Will re-run against other encode sources and increase / decrease rf value as required to hit a target ssim in the future.
jamiemlaw
Veteran User
Posts: 536
Joined: Thu Sep 17, 2009 4:52 pm

Re: Better settings don't always equal better encodes

Post by jamiemlaw »

Don't forget to also include the bitrate in your results.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

jamiemlaw wrote:Don't forget to also include the bitrate in your results.
In the first set of results it is possible to calculate the bitrate (the file was 93 seconds long), but the consensus seemed to be that looking at file sizes wasn't too important. (IE 8x8 increases file size and quality.)

For myself, the coming results are more about the effect of encoding settings upon quality at a given constant quality / rf value, and hence bitrate isn't really important. This is because although I would like to save some hdd room, the most important thing is that (a) the quality of the encode is good enough and (b) the encoding speed is high enough. (For this reason I will be re-running some tests at different rf values if people recommend a different minimum/maximum ssim value.)

Having said that, I will (if I remember!) try to include bitrate information if that's what you want.
jamiemlaw
Veteran User
Posts: 536
Joined: Thu Sep 17, 2009 4:52 pm

Re: Better settings don't always equal better encodes

Post by jamiemlaw »

It's all very well if a particular setting increases the quality and takes less time to encode, but if the file size balloons as a result (it can happen), then maybe it isn't "better" after all. HandBrake's mantra is, "Speed, size, quality… pick two," and so it's good to include all three of those in order to get the big picture (no pun intended).
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

jamiemlaw wrote:It's all very well if a particular setting increases the quality and takes less time to encode, but if the file size balloons as a result (it can happen), then maybe it isn't "better" after all. HandBrake's mantra is, "Speed, size, quality… pick two," and so it's good to include all three of those in order to get the big picture (no pun intended).
Hi again. :)

Primarily I'm interested in quality and speed, though as you can guess from my first post, file size (/ bitrate) is important to me as well. If it weren't I'd simply convert my files to raw video and be done with it. :) Going from my first set of results, I've chosen to look at a range of settings which has (so far) limited the size difference to under 30% from minimum. (Largest encode 38674Kb, smallest 30992.) I'll try to remember to show the bitrate variance, but it might things a bit harder to read. m(_ _)m
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

Anyway...
Once again these results are normalized to make easier reading.
The source file was a 1280*720 copy of Coalgirl's "Ladies Vs Butlers" opening.

For those who are interested, the normalized encode was run at the following settings:

Constant Quality RF=19;
bframes=16:me=dia:8x8dct=0:subq=0:trellis=0

Encoded filesize : 39,601,635 bytes
Encode fps : 22.892128
Mean SSIM : 21.436
Bitrate = 39601635bytes / 92sec = 3362kbps

...
Either my maths is out, or that's high. To me it's still acceptable so long as the quality is fine, but I'd prefer it to be lower.
(Hence my investigation of the other encode settings.)

===========================================
bframes=16:me=dia:8x8dct=0:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.00 === 0.000 === 0.0
1: 1.11 === 0.349 === 6.3
2: 1.41 === 1.000 === 15.6
3: 1.58 === 1.388 === 16.2
4: 1.92 === 1.420 === 17.0
5: 2.26 === 1.419 === 18.7
6: 2.47 === 1.346 === 16.5
7: 2.56 === 1.330 === 17.4
8: 2.86 === 1.434 === 16.7
9: 2.99 === 1.470 === 16.4
--------------------------------------------
10: 6.35 === 1.336 === 24.3 (trellis=2 as per handbrake tooltip)
===========================================
bframes=16:me=hex:8x8dct=0:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.06 === 0.019 === 1.2
1: 1.16 === 0.363 === 7.6
2: 1.38 === 0.961 === 16.5
3: 1.60 === 1.356 === 17.1
4: 1.92 === 1.394 === 18.0
5: 2.25 === 1.390 === 19.6
6: 2.43 === 1.325 === 17.2
7: 2.60 === 1.305 === 18.1
8: 2.94 === 1.406 === 17.3
9: 3.08 === 1.440 === 16.9
--------------------------------------------
10: 6.41 === 1.319 === 24.8 (trellis=2 as per handbrake tooltip)

->
At subq=0, moving to ME=hex takes 6% longer, gains 0.02db and reduces the file size by 1.2%.
At subq=10, this move again takes 6% longer but gives mixed results. SSIM decreases but the file size does so as well.

FOr me, more interesting is that in both cases there is a local maximum in SSIM at subq=4 and subq=9.
The extra work done at subq=5 appears to go towards decreasing file size; and then subq=6 appears to be bad with regard to size and quality.

And subq=10? Going by SSIM and RF alone, I could have encoded this twice at subq=9 and got better quality.
(Of course, SSIM isn't totally accurate, but what is going on here?)

For me, subq=3 looks good for quick encodes, and subq=4/5 looks good for quick but small encodes.

And with 8x8 decimation?...

===========================================
bframes=16:me=dia:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.07 === -0.097 === 0.3
1: 1.12 === 0.245 === 6.6
2: 1.38 === 0.946 === 16.1
3: 1.62 === 1.336 === 17.1
4: 1.96 === 1.372 === 18.0
5: 2.28 === 1.364 === 19.3
6: 2.58 === 1.264 === 17.7
7: 2.65 === 1.242 === 18.8
8: 2.99 === 1.341 === 18.4
9: 3.13 === 1.386 === 17.9
===========================================
bframes=16:me=hex:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.09 === -0.075 === 1.5
1: 1.21 === 0.260 === 7.8
2: 1.47 === 0.913 === 16.9
3: 1.74 === 1.306 === 18.0
4: 2.11 === 1.341 === 18.9
5: 2.51 === 1.335 === 20.2
6: 2.73 === 1.242 === 18.4
7: 2.86 === 1.220 === 19.5
8: 3.11 === 1.315 === 19.0
9: 3.34 === 1.361 === 18.5

->
8x8 decimation uses a bit of cpu, (And decreases SSIM), but again we see the local maximums in SSIM at subq=4 and subq=9.
Again subq=5 beats all but subq=10 in file size reduction whilst giving almost the same SSIM as subq=4.

Now with no-fast-pskip?

===========================================
bframes=16:me=dia:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.09 === -0.061 === -0.3
1: 1.15 === 0.299 === 5.7
2: 1.44 === 1.062 === 14.8
3: 1.67 === 1.421 === 16.1
4: 2.10 === 1.467 === 16.8
5: 2.44 === 1.455 === 18.6
6: 2.58 === 1.325 === 17.3
7: 2.71 === 1.299 === 18.5
8: 3.06 === 1.397 === 18.0
9: 3.20 === 1.445 === 17.6

===========================================
bframes=16:me=hex:subq=(xxx):trellis=0
-------------------------------------------
(xxx): FPS === SSIM Increase === % SIZE REDUCTION
-------------------------------------------
0: 1.12 === -0.037 === 0.8
1: 1.23 === 0.316 === 6.9
2: 1.58 === 1.023 === 15.6
3: 1.80 === 1.384 === 17.0
4: 2.15 === 1.433 === 17.7
5: 2.56 === 1.424 === 19.5
6: 2.76 === 1.298 === 18.0
7: 2.93 === 1.272 === 19.1
8: 3.32 === 1.370 === 18.7
9: 3.48 === 1.419 === 18.2

Again, even slower encodes, larger files and victories for subq=4 and 5.

I know that I'm sample-size-limited; but I've now done 2 sets of encodes with different source files and even if that's nothing statistically, I'm beginning to think that I'll use subq=3,4,5 until someone changes my mind. :) Even if my results don't apply to whatever sources other people are encoding, I think it shows that it's worth doing test encodes to check how much time the encode really has to take.
Last edited by Arlki on Fri Sep 10, 2010 4:12 pm, edited 1 time in total.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

:oops: Oops... Those last results were with no-fast-pskip as well as 8x8 decimation. Sorry about that... m(_ _)m
User avatar
Rodeo
HandBrake Team
Posts: 12757
Joined: Tue Mar 03, 2009 8:55 pm

Re: Better settings don't always equal better encodes

Post by Rodeo »

Arlki wrote:8x8 decimation uses a bit of cpu, (And decreases SSIM), but again we see the local maximums in SSIM at subq=4 and subq=9.
Again subq=5 beats all but subq=10 in file size reduction whilst giving almost the same SSIM as subq=4.

[…]

Again, even slower encodes, larger files and victories for subq=4 and 5.

I know that I'm sample-size-limited; but I've now done 2 sets of encodes with different source files and even if that's nothing statistically, I'm beginning to think that I'll use subq=3,4,5 until someone changes my mind. :) Even if my results don't apply to whatever sources other people are encoding, I think it shows that it's worth doing test encodes to check how much time the encode really has to take.
subq 6 enables psychovisual optimizations, which will improve perceived quality but hurt metrics, thus the lower SSIM. With an encoder like x264, which features plenty of psy opts, you can't trust the metrics to give you the whole picture, at some point you have to look at the output.

Basically, you have to understand what the settings do before you can make valid comparisons, you can't do it the other way around.

The last thing we need to see is another set of uninformed, pointless x264 settings comparison.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

Rodeo wrote:
Arlki wrote:8x8 decimation uses a bit of cpu, (And decreases SSIM), but again we see the local maximums in SSIM at subq=4 and subq=9.
Again subq=5 beats all but subq=10 in file size reduction whilst giving almost the same SSIM as subq=4.

[…]

Again, even slower encodes, larger files and victories for subq=4 and 5.

I know that I'm sample-size-limited; but I've now done 2 sets of encodes with different source files and even if that's nothing statistically, I'm beginning to think that I'll use subq=3,4,5 until someone changes my mind. :) Even if my results don't apply to whatever sources other people are encoding, I think it shows that it's worth doing test encodes to check how much time the encode really has to take.
subq 6 enables psychovisual optimizations, which will improve perceived quality but hurt metrics, thus the lower SSIM. With an encoder like x264, which features plenty of psy opts, you can't trust the metrics to give you the whole picture, at some point you have to look at the output.
Just checking, but you do know that unlike psnr, the ssim approximation that x264 uses is a non-pc / human-orientated metric, don't you?
Regardless; (a) I've already said that I know that ssim isn't the whole picture; (b) the ssim with this source material (animated) drops at 5, not 6.
Basically, you have to understand what the settings do before you can make valid comparisons, you can't do it the other way around.
Please forgive me for misunderstanding "Constant Quality".
Someone mentions ssim, so I look it up, decide that it's appropriate and add it. Now I've learnt something and I believe that despite its' limitations the data that I presented is still useful
The last thing we need to see is another set of uninformed, pointless x264 settings comparison.
Pointless? Have you even bothered to look at any of the other benchmarks in this forum? :shock:
Anyway, I hope you like my next post. :)
User avatar
Rodeo
HandBrake Team
Posts: 12757
Joined: Tue Mar 03, 2009 8:55 pm

Re: Better settings don't always equal better encodes

Post by Rodeo »

Arlki wrote:Just checking, but you do know that unlike psnr, the ssim approximation that x264 uses is a non-pc / human-orientated metric, don't you?
Regardless; (a) I've already said that I know that ssim isn't the whole picture; (b) the ssim with this source material (animated) drops at 5, not 6.
That doesn't mean subq 4 and 5 win. Despite the lower SSIM, there's a strong likeliness that subq 6 and above will look better.
Arlki wrote:
Basically, you have to understand what the settings do before you can make valid comparisons, you can't do it the other way around.
Please forgive me for misunderstanding "Constant Quality".
Someone mentions ssim, so I look it up, decide that it's appropriate and add it. Now I've learnt something and I believe that despite its' limitations the data that I presented is still useful
The data, maybe (if you interpret it correctly). Your conclusions, no. subq 5 o4 4 winning over 6? Have you even looked at the output?

Not to mention you're comparing SSIM across encodes that have a different bitrate. How do you expect to compare compression efficiency that way?

In case you hadn't figured it out, increasing video quality is only a matter of throwing more bits at it (i.e. higher bitrates). The whole point of using slower settings is to get high quality and small file sizes. Putting aside psychovisual optimizations* (which increase perceived quality at the expense of lower metrics), the correct way to measure compression efficiency is to encode for a target bitrate and see which settings give you the highest metrics.

* If you're comparing settings using metrics, you should disable all such optimizations or at least take them out of the equation. Changing a setting that affects psy opts makes metrics pointless, because you can't trust that a higher metric value actually corresponds to a higher perceived quality (even if you're using a reasonably good metric like SSIM). And before you can do that, you have to know which options affect psy opts and which don't.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

Rodeo wrote:
Arlki wrote:Just checking, but you do know that unlike psnr, the ssim approximation that x264 uses is a non-pc / human-orientated metric, don't you?
Regardless; (a) I've already said that I know that ssim isn't the whole picture; (b) the ssim with this source material (animated) drops at 5, not 6.
That doesn't mean subq 4 and 5 win. Despite the lower SSIM, there's a strong likeliness that subq 6 and above will look better.
Have you ever looked at my encodes?
No; but that probably doesn't matter.

Have you ever looked at an encode where subq 4/5 had a higher SSIM than subq=6?
I don't know; but I think that that matters.

Do you think that you're a better judge of image quality than the x264 SSIM approximation?
I'm gessing so, and maybe you are. (And yes, I know ssim isn't exact; and I know it's not the best measurement either; but it's the best that I had.)

I didn't choose 4/5 for quality alone. If quality was all that counted I'd be doing UMH encodes.
Arlki wrote:
Basically, you have to understand what the settings do before you can make valid comparisons, you can't do it the other way around.
Please forgive me for misunderstanding "Constant Quality".
Someone mentions ssim, so I look it up, decide that it's appropriate and add it. Now I've learnt something and I believe that despite its' limitations the data that I presented is still useful
The data, maybe (if you interpret it correctly). Your conclusions, no. subq 5 o4 4 winning over 6? Have you even looked at the output?
[/quote]

Regardless, for me winning or losing in this race involves CPU time.
How can you have not realized that? :shock:
Not to mention you're comparing SSIM across encodes that have a different bitrate. How do you expect to compare compression efficiency that way?
:evil:
I thought my data might be useful but it was done for my own purposes. For me Quality > Speed > Bitrate; so guess which one I'm not really looking at. (Sure it's nice, but it's not my main priority.) Also, efficiency can be measured in different ways, and I'm measuring it against CPU usage, not against storage space. It's like I'm looking at efficiency in moving from a to b in my car by measuring how much petrol it uses, and you're saying "but it takes more tyres than a motorbike!". Storage does matter to me; but not that much. Quality is tops. Speed is second. Your preferences are clearly elsewhere.
User avatar
Rodeo
HandBrake Team
Posts: 12757
Joined: Tue Mar 03, 2009 8:55 pm

Re: Better settings don't always equal better encodes

Post by Rodeo »

Arlki wrote:Have you ever looked at an encode where subq 4/5 had a higher SSIM than subq=6?
I don't know; but I think that that matters.
Yes, of course. In fact, with psy opts enabled, it's likely subq 4 and 5 will always produce streams with a higher SSIM than 6 (all other options being equal, same bitrate). That's because subq 6 enables RDO, which in turn enables psy-RDO, of of x264's most significant psy optimizations. That doesn't mean subq 4/5 will necessarily look better.
Arlki wrote:Do you think that you're a better judge of image quality than the x264 SSIM approximation?
I'm gessing so, and maybe you are. (And yes, I know ssim isn't exact; and I know it's not the best measurement either; but it's the best that I had.)
When psy opts enter the equation, SSIM cannot be trusted blindly. As I said above, psy-RDO will make it so that, at the same bitrate, all other options being equal, subq 6 almost always results in an SSIM lower than subq 4/5. Only you can decide whether the psy opts mean that subq 6 looks better than subq 4/5, but to determine whether you prefer the output with psy-RDO or without, you will actually have to look at the encode, not the metrics.

Now, if I disable all psy opts except AQ (psy=0 will do that), then at the same bitrate, the opposite happens: subq 6 almost always produces a higher SSIM than 4/5. But I don't actually disable psy for my encodes, since despite the lower SSIM, encodes with psy actually look better, at least in my opinion (which is the whole point of the optimizations).
Arlki wrote:For me Quality > Speed > Bitrate; so guess which one I'm not really looking at. (Sure it's nice, but it's not my main priority.) Also, efficiency can be measured in different ways, and I'm measuring it against CPU usage, not against storage space. It's like I'm looking at efficiency in moving from a to b in my car by measuring how much petrol it uses, and you're saying "but it takes more tyres than a motorbike!". Storage does matter to me; but not that much. Quality is tops. Speed is second. Your preferences are clearly elsewhere.
That doesn't mean comparing compression efficiency is useless to your goal. You can find several combinations of x264 settings that will be roughly equally fast, but that doesn't mean they will be equally efficient. You can try every switch until you find some settings that give you the speed that you want: that won't make your settings optimal; there could be another combination of settings that gives you more speed while maintaining a similar quality and file size, or give you better quality while maintaining similar speed and file sizes, or give you smaller files while maintaining similar speed and quality.

Now you're looking at SSIM to determine which options are more useful (and therefore worth spending time on). But you're trying every option before even investigating what options do and in which order they should probably be disabled. For example, 8x8dct is a very useful option with a low speed cost (it has a cost, but there are plenty of other options with a similar cost which are less useful and should be disabled before you disable 8x8dct). Unless using settings much faster than what you've tested so far, enabling or disabling 8x8dct should be dictated by device compatibility. Sure, it increases file size, but if you adjust the RF to match the file size on an encode with it disabled, and there's a strong likeliness that the 8x8dct encode will have a higher metric and better quality (it's not a psy opt).
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

Rodeo wrote:
Arlki wrote:Have you ever looked at an encode where subq 4/5 had a higher SSIM than subq=6?
I don't know; but I think that that matters.
Yes, of course. In fact, with psy opts enabled, it's likely subq 4 and 5 will always produce streams with a higher SSIM than 6 (all other options being equal, same bitrate). That's because subq 6 enables RDO, which in turn enables psy-RDO, of of x264's most significant psy optimizations. That doesn't mean subq 4/5 will necessarily look better.
Do you have any benchmarks to back up your assertion that SSIM will likely be lower at the same bitrate if psy-opts are enabled?
If you look at my new thread (Athlon x2 3800+ - 186 benchmarks @ diff settings) I think you'll see that the opposite (psy-opts + lower bitrate = higher ssim) is often true.
Arlki wrote:Do you think that you're a better judge of image quality than the x264 SSIM approximation?
I'm gessing so, and maybe you are. (And yes, I know ssim isn't exact; and I know it's not the best measurement either; but it's the best that I had.)
When psy opts enter the equation, SSIM cannot be trusted blindly. As I said above, psy-RDO will make it so that, at the same bitrate, all other options being equal, subq 6 almost always results in an SSIM lower than subq 4/5. Only you can decide whether the psy opts mean that subq 6 looks better than subq 4/5, but to determine whether you prefer the output with psy-RDO or without, you will actually have to look at the encode, not the metrics.

Now, if I disable all psy opts except AQ (psy=0 will do that), then at the same bitrate, the opposite happens: subq 6 almost always produces a higher SSIM than 4/5. But I don't actually disable psy for my encodes, since despite the lower SSIM, encodes with psy actually look better, at least in my opinion (which is the whole point of the optimizations).
Have you actually tried this, or are you guessing that's what will happen?
Arlki wrote:For me Quality > Speed > Bitrate; so guess which one I'm not really looking at. (Sure it's nice, but it's not my main priority.) Also, efficiency can be measured in different ways, and I'm measuring it against CPU usage, not against storage space. It's like I'm looking at efficiency in moving from a to b in my car by measuring how much petrol it uses, and you're saying "but it takes more tyres than a motorbike!". Storage does matter to me; but not that much. Quality is tops. Speed is second. Your preferences are clearly elsewhere.
That doesn't mean comparing compression efficiency is useless to your goal. You can find several combinations of x264 settings that will be roughly equally fast, but that doesn't mean they will be equally efficient. You can try every switch until you find some settings that give you the speed that you want: that won't make your settings optimal; there could be another combination of settings that gives you more speed while maintaining a similar quality and file size, or give you better quality while maintaining similar speed and file sizes, or give you smaller files while maintaining similar speed and quality.
Umm... Why do you think I tested so many settings?
Now you're looking at SSIM to determine which options are more useful (and therefore worth spending time on). But you're trying every option before even investigating what options do and in which order they should probably be disabled. For example, 8x8dct is a very useful option with a low speed cost (it has a cost, but there are plenty of other options with a similar cost which are less useful and should be disabled before you disable 8x8dct). Unless using settings much faster than what you've tested so far, enabling or disabling 8x8dct should be dictated by device compatibility. Sure, it increases file size, but if you adjust the RF to match the file size on an encode with it disabled, and there's a strong likeliness that the 8x8dct encode will have a higher metric and better quality (it's not a psy opt).
Do you think that I didn't look at this given the handbrake tooltip? The only reason I haven't looked at it with ssim yet was because my ssim follow-up was aimed at improving a set of benchmarks performed without it, so it was quickest to leave it out. You'll see that I have already looked at this with new source materials and found that it normally results in lower bitrate / higher ssim and higher cpu.
User avatar
Rodeo
HandBrake Team
Posts: 12757
Joined: Tue Mar 03, 2009 8:55 pm

Re: Better settings don't always equal better encodes

Post by Rodeo »

Arlki wrote:Do you have any benchmarks to back up your assertion that SSIM will likely be lower at the same bitrate if psy-opts are enabled?
If you look at my new thread (Athlon x2 3800+ - 186 benchmarks @ diff settings) I think you'll see that the opposite (psy-opts + lower bitrate = higher ssim) is often true.
AFAICT, the only unfixed option in your benchmarks that affects psy is subq. As expected, I can see a slight drop in SSIM and a small increase in file size comparing subq 5 to 6 (where psy-RDO kicks in) in many cases:
Arlki wrote:

Code: Select all

Encode Name : <fps1, ssim1, size1> <fps2, ssim2, size2> <fps3, ssim3, size3>
------------------------------------------------------------------------------------------------------------------
[…]
1 0 Dia 5: <38.976, 19.464, 22502994> <37.030, 19.757, 12293160> <39.547, 16.958, 19140617>
1 0 Dia 6: <32.543, 19.415, 22625820> <32.101, 19.641, 12343081> <33.445, 16.923, 19413161>
[…]
1 0 Hex 5: <38.535, 19.452, 22445090> <36.956, 19.767, 12270201> <38.310, 16.952, 19084031>
1 0 Hex 6: <31.581, 19.403, 22564391> <30.941, 19.651, 12326597> <32.586, 16.921, 19379998>
[…]
1 1 Dia 5: <39.699, 19.779, 20921267> <37.513, 19.919, 12177313> <38.455, 17.411, 18974965>
1 1 Dia 6: <30.240, 19.785, 20442999> <29.639, 19.797, 11844621> <31.247, 17.508, 19028174>
[…]
1 1 Hex 5: <36.407, 19.765, 20887486> <34.610, 19.928, 12154272> <35.331, 17.407, 18945934>
1 1 Hex 6: <29.106, 19.769, 20416264> <28.571, 19.807, 11838701> <30.216, 17.506, 19011139>
[…]
2 0 Dia 5: <29.569, 19.463, 22269957> <28.499, 19.787, 11996168> <29.846, 16.918, 18896959>
2 0 Dia 6: <25.275, 19.435, 22435708> <24.870, 19.686, 12125022> <26.075, 16.902, 19234436>
[…]
2 0 Hex 5: <28.446, 19.452, 22204730> <27.513, 19.795, 11982801> <28.890, 16.914, 18853927>
2 0 Hex 6: <24.463, 19.421, 22372976> <24.011, 19.696, 12113783> <25.331, 16.899, 19198916>
[…]
2 1 Dia 5: <29.110, 19.781, 20843266> <27.803, 19.947, 11928870> <28.737, 17.371, 18812497>
2 1 Dia 6: <23.769, 19.801, 20419330> <22.412, 19.833, 11697081> <24.287, 17.490, 18933358>
[…]
2 1 Hex 5: <28.558, 19.766, 20795445> <27.262, 19.957, 11914361> <28.266, 17.367, 18781098>
2 1 Hex 6: <23.499, 19.788, 20385155> <22.950, 19.844, 11693731> <24.243, 17.488, 18916783>
[…]
3 0 Dia 5: <26.815, 19.461, 22271262> <25.966, 19.786, 11996826> <27.013, 16.909, 18865805>
3 0 Dia 6: <23.279, 19.435, 22435860> <22.883, 19.689, 12118733> <23.785, 16.898, 19201942>
[…]
3 0 Hex 5: <25.845, 19.453, 22209968> <24.958, 19.798, 11980151> <26.217, 16.906, 18822369>
3 0 Hex 6: <22.481, 19.427, 22375701> <22.091, 19.701, 12105611> <23.219, 16.894, 19165156>
[…]
3 1 Dia 5: <26.420, 19.778, 20854203> <25.260, 19.949, 11924680> <25.987, 17.362, 18788740>
3 1 Dia 6: <21.919, 19.803, 20432216> <21.496, 19.836, 11699297> <22.326, 17.487, 18917156>
[…]
3 1 Hex 5: <25.939, 19.766, 20819529> <24.742, 19.958, 11913841> <25.732, 17.359, 18758440>
3 1 Hex 6: <21.773, 19.787, 20395827> <21.145, 19.850, 11704505> <22.230, 17.482, 18898652>
You're right that it's not always true, however (sometimes file size decreases or SSIM increases or both).
Arlki wrote:
Arlki wrote:Do you think that you're a better judge of image quality than the x264 SSIM approximation?
I'm gessing so, and maybe you are. (And yes, I know ssim isn't exact; and I know it's not the best measurement either; but it's the best that I had.)
When psy opts enter the equation, SSIM cannot be trusted blindly. As I said above, psy-RDO will make it so that, at the same bitrate, all other options being equal, subq 6 almost always results in an SSIM lower than subq 4/5. Only you can decide whether the psy opts mean that subq 6 looks better than subq 4/5, but to determine whether you prefer the output with psy-RDO or without, you will actually have to look at the encode, not the metrics.

Now, if I disable all psy opts except AQ (psy=0 will do that), then at the same bitrate, the opposite happens: subq 6 almost always produces a higher SSIM than 4/5. But I don't actually disable psy for my encodes, since despite the lower SSIM, encodes with psy actually look better, at least in my opinion (which is the whole point of the optimizations).
Have you actually tried this, or are you guessing that's what will happen?
I have tried it.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: Better settings don't always equal better encodes

Post by jbrjake »

Arlki wrote:
Rodeo wrote:
Arlki wrote:Have you ever looked at an encode where subq 4/5 had a higher SSIM than subq=6?
I don't know; but I think that that matters.
Yes, of course. In fact, with psy opts enabled, it's likely subq 4 and 5 will always produce streams with a higher SSIM than 6 (all other options being equal, same bitrate). That's because subq 6 enables RDO, which in turn enables psy-RDO, of of x264's most significant psy optimizations. That doesn't mean subq 4/5 will necessarily look better.
Do you have any benchmarks to back up your assertion that SSIM will likely be lower at the same bitrate if psy-opts are enabled?
<snip>
Have you actually tried this, or are you guessing that's what will happen?
It's not like Rodeo is making this stuff up. Anyone who spends even a modicum of time researching x264 options can find this stuff out. All you have to do is google. The developers of x264 communicate these things very clearly.

If I had had any idea you would do something as foolish as leave psy-opt on when doing metric comparisons I would never have even mentioned SSIM to you.

I'm not sure why you're dedicating so much time and effort to these fruitless empirical tests. x264's developers have a much better understanding of how encoder features interact than any of us do, and it would behoove you to research what they have to say. x264 is an open source project, not a black box that can only be described by statistics based on questionable metrics.
Arlki
Posts: 25
Joined: Thu Sep 09, 2010 7:09 am

Re: Better settings don't always equal better encodes

Post by Arlki »

jbrjake wrote:Anyone who spends even a modicum of time researching x264 options can find this stuff out. All you have to do is google. The developers of x264 communicate these things very clearly.
I would ask how you found that post using google, but that example is atrocious anyway. Looking at that post, the only things clearly communicated in there are the bad formatting and readability of the original post as well as the attitude of dark_shikari; though there may be a history that causes him to respond in such a way. If that is your idea of clear communication then I think you might want to look at these and then re-evaluate that position.

http://www.mplayerhq.hu/DOCS/HTML/en/me ... ng-options

http://mewiki.project357.com/wiki/X264_Settings

These pages, along with one called "DeathTheSheep", which now may not be updated and others which I've lost; are what I based my own initial encode settings upon.
You'll notice, however, that even though they state things clearly; they give very little way to tell even approximately how much one setting will decrease the encoding speed for a machine whose architecture (socket 939) and speed matches mine - or in fact probably most other people's. Often things are stated as "setting x tends to decrease encode speed at high values of y", but that's not sufficient for me to work out what sort of fps I should be getting at each setting. (At least beyond "well, it should be slower".) Since I am cpu-time constrained, that is very important to me; and so I set about doing these benchmarks.
jbrjake wrote:If I had had any idea you would do something as foolish as leave psy-opt on when doing metric comparisons I would never have even mentioned SSIM to you.
To be honest, though I replied politely and advised you that I was aware that snr and ssim and so on measurements weren't the whole story; I viewed your entire comment as a sarcastic snipe meant to demonstrate how great your encoding knowledge was. The fact that you at no time responded with the above information when I expressed my ignorance, nor offered in fact anything constructive, does nothing to make me change that opinion.I hope you will notice, however, that regardless of my opinion, I did look at ssim, and that I concentrated upon the lower subq settings in order to increase the accuracy and relevancy of my investigation.
jbrjake wrote:I'm not sure why you're dedicating so much time and effort to these fruitless empirical tests.
Can't you work it out? Have I not made it clear that I want to encode things quickly but with as much quality as possible within that time? Can you not see how I am now able to predict roughly how long an encode will take on my pc, (and on future pcs, once I establish the scaling factor) no matter how I change the settings? (At least for a particular source type - but that's why I've tested for different types of source material as well. (IE SD live-action and HD animated.)
jbrjake wrote:x264's developers have a much better understanding of how encoder features interact than any of us do, and it would behoove you to research what they have to say. x264 is an open source project, not a black box that can only be described by statistics based on questionable metrics.
And so you present another page that gives no way to work out within any reasonable limit just how quickly an encode will run on my machine or a machine of known better / worse performance.

Before assuming that I have done no research, why not look at some of my settings and wonder if maybe I didn't pull those out of thin air. Beyond that, if you have something constructive or helpful to offer then I would love to hear it; though I'm afraid that if I do as suggested I'll only be lining myself up for another sarcy comment.

1. I am aware that SSIM is not the best measure of image quality.

2. I am aware that the SSIM used by x264 is only an approximation to the real thing.

3. I still have no other scientific way to quantify either the quality that a group of people would assign to a specific (high bitrate) encode; nor do I know any way to find out from x264 exactly what quality it thinks it has achieved. For that reason you'll notice that I stopped at subq=6 in my latest set of benchmarks (where I believe the psy-opts kick in in full) as it didn't make sense to spend longer where psy-opts might have an even greater effect on the results. Until I can quantify the increase in perceived quality, I will keep my analysis to the lower subq values (1-5) where my current pc tends encode at an okay framerate even if I increase other settings.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: Better settings don't always equal better encodes

Post by jbrjake »

It's pretty silly to ignore the advice of the lead developer of x264 (in the form of a plea to use the preset system he spent so long tuning and in the form of an option-by-option list describing what features do and what the best settings are for them) in favor of third-party resources, some of which are horrifically out of date. Jason has an attitude (pot, kettle, etc), everyone knows that, but he's also a genius. Spurning his words because you dislike doom9's formatting and his tone is cutting off your nose to spite your face.

It's impossible to identify a "scaling factor" for how x264 is going to perform on potential future PCs. Why? Because x264 is always getting its existing code optimized optimized, processors are always coming out with new instruction set extensions, and x264 is enhanced to take advantage of the extensions.

I'm tried suggesting this gently but maybe it's time to be less subtle:

If you don't understand x264's options, the best way to learn how they interact regarding time and quality isn't to waste days and days of your time doing empirical testing, but rather to research what the x264 developers say on the matter in the forms of their public statements about the options and, perhaps more useful for you, their new preset system.
TedJ
Veteran User
Posts: 5388
Joined: Wed Feb 20, 2008 11:25 pm

Re: Better settings don't always equal better encodes

Post by TedJ »

jbrjake wrote:It's impossible to identify a "scaling factor" for how x264 is going to perform on potential future PCs. Why? Because x264 is always getting its existing code optimized optimized, processors are always coming out with new instruction set extensions, and x264 is enhanced to take advantage of the extensions.
+1

Speaking from personal experience, enhancements and optimisations to libx264 over the last 18-24 months have netted me an approximate 18-20% performance boost using similar settings on the same hardware.
Post Reply