varying threads has no effect (--encopts threads=x)

[graysky]

I've been looking at encoding efficiency as a function of threads using the following. What I'm noticing is that on my quad core (no hyperthreads), values >8 have no effect of throughput, encode time, or video quality. This cannot be... can it?

threads

Default: auto (frame based threads: 1.5 * logical processors, rounded down; slice based threads: 1 * logical processors)

Enables parallel encoding by using more than 1 thread to increase speed on multi-core systems. The quality loss from multiple threads is mostly negligible unless using very high numbers of threads (say, above 16). The speed gain should be slightly less than linear until you start using more than 1 thread per 40px of vertical video, at which point the gain from additional threads sharply decreases.

x264 currently has an internal limit on the number of threads set at 128, realistically you should never set it this high.

For example, I used a value of 6, 16, 32, 128 and all metrics were the same. Am I not calling the threads switch correctly?

Code: Select all

$ HandBrakeCLI --input 23sec-720p60.m2ps --no-dvdnav --audio none --crop 0:0:0:0 --preset=Normal --output tit.mp4  --encopts threads=10 2> encode.log

Example:

Threads/Encode Time
6 / 48 sec
8 / 44 sec
16 / 44 sec
32 / 44 sec
128 / 44 sec

Code: Select all

[15:29:36] hb_init: starting libhb thread
HandBrake svn3853 (2011040401) - Linux x86_64 - http://handbrake.fr
4 CPUs detected
Opening 23sec-720p60.m2ps...
[15:29:36] hb_scan: path=23sec-720p60.m2ps, title_index=1
libbluray/bdnav/index_parse.c:157: indx_parse(): error opening 23sec-720p60.m2ps/BDMV/index.bdmv
libbluray/bluray.c:1376: nav_get_title_list(23sec-720p60.m2ps) failed (0x1d76db0)
[15:29:36] bd: not a bd - trying as a stream/file instead
libdvdread: Using libdvdcss version 1.2.10 for DVD access
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
ERROR: dvd: ifoOpen failed
[15:29:36] file is MPEG DVD Program Stream
[15:29:36] add_audio_to_title: added AC3 audio stream 0x80bd
[15:29:36] scan: decoding previews for title 1
[15:29:36] scan: audio 0x80bd: AC-3, rate=48000Hz, bitrate=384000 Unknown (AC3) (2.0 ch)
[15:29:36] scan: 10 previews, 1280x720, 59.940 fps, autocrop = 0/2/0/2, aspect 16:9, PAR 1:1
[15:29:36] scan: title (0) job->width:1280, job->height:720
[15:29:36] libhb: scan thread found 1 valid title(s)
+ title 1:
  + stream: 23sec-720p60.m2ps
  + duration: 00:00:22
  + size: 1280x720, pixel aspect: 1/1, display aspect: 1.78, 59.940 fps
  + autocrop: 0/2/0/2
  + chapters:
    + 1: cells 0->0, 0 blocks, duration 00:00:22
  + audio tracks:
    + 1, Unknown (AC3) (2.0 ch) (iso639-2: und), 48000Hz, 384000bps
  + subtitle tracks:
+ Using preset: Normal
Audio codecs and no valid audio tracks, skipping codec faac
Ignoring sample rate 0, no audio tracks
Ignoring mixdown, no audio tracks
Ignoring bitrate 160, no audio tracks
Ignoring drc, no audio tracks
[15:29:36] 1 job(s) to process
[15:29:36] starting job
[15:29:36] sync: expecting 1414 video frames
[15:29:36] work: only 1 chapter, disabling chapter markers
[15:29:36] job configuration:
[15:29:36]  * source
[15:29:36]    + 23sec-720p60.m2ps
[15:29:36]    + title 1, chapter(s) 1 to 1
[15:29:36]  * destination
[15:29:36]    + tit.mp4
[15:29:36]    + container: MPEG-4 (.mp4 and .m4v)
[15:29:36]  * video track
[15:29:36]    + decoder: mpeg2
[15:29:36]      + bitrate 18575 kbps
[15:29:36]    + frame rate: same as source (around 59.940 fps)
[15:29:36]    + strict anamorphic
[15:29:36]      + storage dimensions: 1280 * 720 -> 1280 * 720, crop 0/0/0/0, mod 0
[15:29:36]      + pixel aspect ratio: 1 / 1
[15:29:36]      + display dimensions: 1280 * 720
[15:29:36]    + encoder: x264
[15:29:36]      + options: threads=10
[15:29:36]      + quality: 20.00 (RF)
[15:29:36] reader: first SCR 146 id 224 DTS 13019
[15:29:36] encx264: min-keyint: 60, keyint: 600
[15:29:36] encx264: encoding with stored aspect 1/1
[15:29:36] encx264: Encoding at constant RF 20.000000
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
[15:29:36] mpeg2: "" (1) at frame 0 time 4504
x264 [info]: profile High, level 3.2
[15:29:36] sync: video time didn't advance - dropped 1 frames (delta 16 ms, current 21014, next 21021, dur 7)
[15:30:17] reader: done. 0 scr changes
[15:30:20] work: average encoding speed for job is 33.035583 fps
[15:30:20] mux: track 0, 1374 frames, 27299855 bytes, 9513.69 kbps, fifo 8
[15:30:20] sync: got 1374 frames, 1414 expected
[15:30:20] mpeg2 done: 1375 frames
[15:30:20] render: lost time: 0 (0 frames)
[15:30:20] render: gained time: 0 (0 frames) (0 not accounted for)
x264 [info]: frame I:8     Avg QP:20.26  size: 88748  PSNR Mean Y:46.06 U:48.40 V:50.06 Avg:46.87 Global:46.81
x264 [info]: frame P:690   Avg QP:23.30  size: 28892  PSNR Mean Y:42.87 U:45.71 V:47.90 Avg:43.79 Global:43.61
x264 [info]: frame B:676   Avg QP:23.83  size:  9844  PSNR Mean Y:42.38 U:45.43 V:48.06 Avg:43.35 Global:43.17
x264 [info]: consecutive B-frames: 19.5% 39.6% 15.3% 25.6%
x264 [info]: mb I  I16..4:  3.2% 92.1%  4.7%
x264 [info]: mb P  I16..4:  1.6% 17.3%  0.3%  P16..4: 43.8% 16.9%  9.4%  0.0%  0.0%    skip:10.7%
x264 [info]: mb B  I16..4:  0.1%  0.8%  0.0%  B16..8: 40.1%  4.4%  1.3%  direct: 6.4%  skip:46.9%  L0:39.5% L1:53.7% BI: 6.8%
x264 [info]: 8x8 transform intra:90.0% inter:83.9%
x264 [info]: coded y,uvDC,uvAC intra: 80.0% 80.7% 20.2% inter: 33.8% 32.6% 2.6%
x264 [info]: i16 v,h,dc,p: 22% 19% 16% 44%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 17% 29%  6%  5%  5%  6%  8%  9%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 21% 15%  8%  8%  7%  8%  8%  7%
x264 [info]: i8c dc,h,v,p: 53% 22% 20%  5%
x264 [info]: Weighted P-Frames: Y:0.9% UV:0.1%
x264 [info]: ref P L0: 58.6% 16.8% 17.1%  7.4%  0.1%
x264 [info]: ref B L0: 85.0% 14.2%  0.8%
x264 [info]: ref B L1: 94.6%  5.4%
x264 [info]: SSIM Mean Y:0.9745838 (15.949db)
x264 [info]: PSNR Mean Y:42.647 U:45.588 V:47.991 Avg:43.592 Global:43.399 kb/s:9517.27
[15:30:20] libhb: work result = 0

Rip done!
HandBrake has exited.

[Deleted User 11865]

You only have 4 cores. Threads are set to 1.5 * cpu_count by default for a reason.

[graysky]

You only have 4 cores. Threads are set to 1.5 * cpu_count by default for a reason.

I understand that, but I'm interesting in seeing how changing the value of 6 to a higher one will affect encode time, quality, etc. It's unclear to me what I'm doing wrong with my switch setting. Whether it's "good" or not, I should see an affect, no?

[jamiemlaw]

Imagine the task of hammering a nail into a wall. Two men are assigned to the task, but they only have one hammer. The job won't go any faster just because there are two workers, because they still only have the one hammer. They can take turns if they wish, but the job doesn't get done any quicker as a result.

Same with thread and cores.

[Deleted User 11865]

You only have 4 cores. Threads are set to 1.5 * cpu_count by default for a reason.

I understand that, but I'm interesting in seeing how changing the value of 6 to a higher one will affect encode time, quality, etc. It's unclear to me what I'm doing wrong with my switch setting. Whether it's "good" or not, I should see an affect, no?

More threads will reduce encode time up to a point. The default number of threads should give you (almost) as much speed as you're going to get (as you can see, 8 threads instead of 6 give you a small boost, but more threads don't help).

As for the quality difference, it should be negligible under most circumstances.

[graysky]

You only have 4 cores. Threads are set to 1.5 * cpu_count by default for a reason.

I understand that, but I'm interesting in seeing how changing the value of 6 to a higher one will affect encode time, quality, etc. It's unclear to me what I'm doing wrong with my switch setting. Whether it's "good" or not, I should see an affect, no?

More threads will reduce encode time up to a point. The default number of threads should give you (almost) as much speed as you're going to get (as you can see, 8 threads instead of 6 give you a small boost, but more threads don't help).

As for the quality difference, it should be negligible under most circumstances.

You'd think that, but according to my tests, there is no statically significant difference encode speed wise between threads of 6-64 using kernel 2.6.38.2 with the ck3 patchset (Brain [Censored] Scheduler v0.400):


Y-axis is total encode time; average of three runs per thread (x264-t06 is --threads=6 and x264-t12 is --threads=12, etc.).

[Deleted User 11865]

You'd think that, but according to my tests, there is no statically significant difference encode speed wise between threads of 6-64 using kernel 2.6.38.2 with the ck3 patchset (Brain [Censored] Scheduler v0.400):

I'm sorry; I don't follow. What's your point?

[graysky]

I'm sorry; I don't follow. What's your point?

Threads=6 is statistically identical to threads=64 with regard to overall encode speed. In other words, overloading threads does not seem to affect efficiency. This is counterintuitive.

Here is how you read that boxplot:
http://img824.imageshack.us/img824/1856 ... rcles1.png
http://img850.imageshack.us/img850/5746 ... rcles2.png
http://img402.imageshack.us/img402/1710 ... rcles3.png

[Deleted User 11865]

I'm sorry; I don't follow. What's your point?

Threads=6 is statistically identical to threads=64 with regard to overall encode speed.

Again, this is surprising, how? OMG, there a point where more threads don't help! Big deal, this is expected.
Regarding the other possible issue, x264's handling of large thread pools must be good enough that the effect of large thread counts on encoding speeds is insignificant (i.e. no statistically significant slowdown).

[JohnAStebbins]

Just because you tell x264 it *can* use X threads does not mean it *will* use X threads. There is an absolute limit to how many it will create that is somewhere around picture height / 40.