Different file sizes in same encodings on same machine?

Support for HandBrake on Linux, Solaris, and other Unix-like platforms
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
zak
Posts: 17
Joined: Tue Jun 02, 2009 5:33 pm

Different file sizes in same encodings on same machine?

Post by zak »

Description of problem or question:
Not really a problem (I hope) but I would like clarity about this:
After encoding the same video with the same command line execution on the same machine (Intel Core i7-12700) with the same HandBrake version the resulting file sizes slightly differ every time. This did not happen on my old machine with an Intel Xeon E3-1240L.

b_deterministic is set to 1 in my source:

Code: Select all

build/contrib/x264/x264-snapshot-20210613-3065/common/base.c:    param->b_deterministic = 1;
Thank you for your work!


Steps to reproduce the problem (If Applicable):
1. start the encoding, for example:

Code: Select all

"/usr/local/bin/HandBrakeCLI" --input="VIDEO/" --title=1 --output="test.mkv" --markers --encoder=x264 --encoder-preset=veryslow --quality=20 --audio=4,3,4,2,1,2 --aencoder=ffac3,copy:dts,copy:dtshd,ffac3,copy:dts,copy:dtshd --ab=640,,,640,, --crop=0:0:64:64 --auto-anamorphic --decomb --subtitle=1,2,1,2 --subtitle-forced=3,4
2. Move the file to another name, e.g. "mv test.mkv test1.mkv"
3. start the same encoding again.
4. Compare file sizes.


HandBrake version (e.g., 1.0.0):
HandBrake 1.5.1, compiled from source while activating MODULES += contrib/x264 in make/include/main.defs

Operating system and version (e.g., Ubuntu 16.04 LTS, macOS 10.13 High Sierra, Windows 10 Creators Update):
Debian GNU/Linux 11 (bullseye)

(with latest updates installed of course)

HandBrake Activity Log ***required*** (see How-to get an activity log)

https://pastebin.com/UNq6zYMZ
rollin_eng
Veteran User
Posts: 4478
Joined: Wed May 04, 2011 11:06 pm

Re: Different file sizes in same encodings on same machine?

Post by rollin_eng »

A small difference is expected, can you post 2 logs so we can see the difference.
zak
Posts: 17
Joined: Tue Jun 02, 2009 5:33 pm

Re: Different file sizes in same encodings on same machine?

Post by zak »

Another log: https://pastebin.com/tgZRQ0VL

As I said it did not happen with my other CPU. Would there be a way to create completely identical video files? (not counting meta-data of course)
zak
Posts: 17
Joined: Tue Jun 02, 2009 5:33 pm

Re: Different file sizes in same encodings on same machine?

Post by zak »

As far as I see the only relevant difference is in this line:

Code: Select all

[19:20:00] mux: track 0, 139205 frames, 6800437243 bytes, 9370.07 kbps, fifo 512
[22:12:01] mux: track 0, 139205 frames, 6800436109 bytes, 9370.07 kbps, fifo 512
Comparing with further logs I see that it is always this line, i.e. track 0. This applies to another video I tested as well.
User avatar
Rodeo
HandBrake Team
Posts: 12986
Joined: Tue Mar 03, 2009 8:55 pm

Re: Different file sizes in same encodings on same machine?

Post by Rodeo »

That's a little odd indeed, but it won't be easy to figure out where the nondeterminism comes from.

Is the issue reproducible on a shorter samples (e.g. encoding the same chapter, say, #12, multiple times with the same HandBrake instance)?
rollin_eng
Veteran User
Posts: 4478
Joined: Wed May 04, 2011 11:06 pm

Re: Different file sizes in same encodings on same machine?

Post by rollin_eng »

Isn't x264 non-deterministic? Thus small differences are normal?
mduell
Veteran User
Posts: 7757
Joined: Sat Apr 21, 2007 8:54 pm

Re: Different file sizes in same encodings on same machine?

Post by mduell »

x264 is non-deterministic with multithreading and VBV.
zak
Posts: 17
Joined: Tue Jun 02, 2009 5:33 pm

Re: Different file sizes in same encodings on same machine?

Post by zak »

I just did some tests and encoded chapter 12 three times and then chapter 1 two times (just added -c12 resp. -c1 to the command line)
All the respective files were identical. (I did a "mkvextract tracks test.mkv 0:test.x264" and compared those files using "diff".)

Maybe the differences don't happen that often and only on longer files?
I will probably do some more tests later...
User avatar
Rodeo
HandBrake Team
Posts: 12986
Joined: Tue Mar 03, 2009 8:55 pm

Re: Different file sizes in same encodings on same machine?

Post by Rodeo »

mduell wrote: Thu Sep 15, 2022 4:14 pm x264 is non-deterministic with multithreading and VBV.
s/x264/x265 (hence why x265 does or did print a warning to that effect when you enable VBV)

x264 is deterministic on my end at least.
mduell
Veteran User
Posts: 7757
Joined: Sat Apr 21, 2007 8:54 pm

Re: Different file sizes in same encodings on same machine?

Post by mduell »

x264 dev mailing list says it's the intended behavior https://mailman.videolan.org/pipermail/ ... 11036.html
User avatar
Rodeo
HandBrake Team
Posts: 12986
Joined: Tue Mar 03, 2009 8:55 pm

Re: Different file sizes in same encodings on same machine?

Post by Rodeo »

Fair enough :-)
Post Reply