I ran some more tests (CRF 18.5,
veryfast, high, level 4.1, tune film) on a Haswell machine:
Code: Select all
Th/LA cpu fps size
2x10x2 40/1 15% 83.138443 367,969,283
40/2 20% 89.687965 368,693,090
40/4 25% 105.647507 372,379,622
40/10 35% 140.687973 381,441,796
40/20 40% 167.331696 386,208,589
40/40 40% 149.477509 386,198,677
40/60 40% 174.871857 386,172,894
40/80 40% 172.604691 386,191,572
There are two physical CPUs, each with 10 cores (HyperThreaded).
Lookahead thread is indeed the first limiter. But there are three problems with using more lookahead threads: (i) it is still not able to saturate the CPU, (ii) encoding efficiency goes down, (iii) 2-CPU overhead [not apparent here].
How about using just one CPU?
Code: Select all
Th/LA cpu fps size
1x10x1 10/1 55% 80.206764 362,821,037 # using one set of cores
1x10x2 20/1 30% 72.592583 368,378,414 # w/HT
1x10x2 20/2 40% 86.008881 369,142,986 # w/HT
(CPU% is for the active CPUs only.)
We need 2 lookahead threads to feed 20 processors, but size goes up. Even with one lookahead thread, size depends on the number of threads. 20 will use more space than 10.
And lastly, how about going slow:
Code: Select all
Th/LA cpu fps size
1x 1x1 1/1 100% 22.973820 362,678,693
1x 1x2 1/1 70% 24.140602 362,678,693 # using one core w/HT
1x 2x1 1/1 70% 27.276686 362,678,693 # using two cores
1x 2x2 4/1 60% 46.623272 362,610,642 # using two cores w/HT
1x 3x2 6/1 40% 60.376286 362,628,534
(CPU% is for the active CPUs only.)
Surprisingly, using several encoding threads is more space efficient than single-threading, but it is very slight and could be specific to this video.
Basically, the takeaway for large multi-core machine:
- Encode in parallel
- Use just one lookahead thread for the smallest size
- Use 4-8 threads per encoding for maximum CPU efficiency
- Don't use cores from different CPUs on the same encoding [not apparent from the results here]
Log for 40 threads / 10 lookahead threads:
Code: Select all
[17:10:43] hb_init: starting libhb thread
[17:10:43] thread 7f9457526700 started ("libhb")
HandBrake 1.0.3 (2017032400) - Linux x86_64 - https://handbrake.fr
40 CPUs detected
Opening bbb_sunflower_1080p_30fps_normal.mp4...
[17:10:43] CPU: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
[17:10:43] - Intel microarchitecture Haswell
[17:10:43] - logical processor count: 40
[17:10:43] hb_scan: path=bbb_sunflower_1080p_30fps_normal.mp4, title_index=1
udfread ERROR: ECMA 167 Volume Recognition failed
disc.c:274: failed opening UDF image bbb_sunflower_1080p_30fps_normal.mp4
disc.c:352: error opening file BDMV/index.bdmv
disc.c:352: error opening file BDMV/BACKUP/index.bdmv
[17:10:43] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread: Encrypted DVD support unavailable.
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdread:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[17:10:43] dvd: not a dvd - trying as a stream/file instead
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_30fps_normal.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isomavc1
creation_time : 2013-12-16 17:44:39
title : Big Buck Bunny, Sunflower version
artist : Blender Foundation 2008, Janus Bager Kristensen 2013
comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
genre : Animation
composer : Sacha Goedegebure
Duration: 00:10:34.60, start: 0.000000, bitrate: 3481 kb/s
Stream #0:0(und): Video: h264 (High) [avc1 / 0x31637661]
yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 2998 kb/s
30 fps, 30k tbn (default)
Metadata:
creation_time : 2013-12-16 17:44:39
handler_name : GPAC ISO Video Handler
Stream #0:1(und): Audio: mp3 [mp4a / 0x6134706D]
48000 Hz, 2 channels, s16p, 160 kb/s (default)
Metadata:
creation_time : 2013-12-16 17:44:42
handler_name : GPAC ISO Audio Handler
Stream #0:2(und): Audio: ac3 [ac[45]3 / 0x332D6361]
48000 Hz, 5.1, fltp, 320 kb/s (default)
Metadata:
creation_time : 2013-12-16 17:44:42
handler_name : GPAC ISO Audio Handler
Side data:
audio service type: main
[17:10:43] scan: decoding previews for title 1
[17:10:43] scan: audio 0x1: mp3, rate=48000Hz, bitrate=160000 Unknown (MP3) (2.0 ch)
[17:10:43] scan: audio 0x2: ac3, rate=48000Hz, bitrate=320000 Unknown (AC3) (5.1 ch)
Scanning title 1 of 1, preview 4, 40.00 %
Scanning title 1 of 1, preview 10, 100.00 %[17:10:44] scan: 10 previews, 1920x1080, 30.000 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 1:1
[17:10:44] libhb: scan thread found 1 valid title(s)
+ Using preset: CLI Default
+ title 1:
+ stream: bbb_sunflower_1080p_30fps_normal.mp4
+ duration: 00:10:34
+ size: 1920x1080, pixel aspect: 1/1, display aspect: 1.78, 30.000 fps
+ autocrop: 0/0/0/0
+ support opencl: no
+ chapters:
+ 1: cells 0->0, 0 blocks, duration 00:10:34
+ audio tracks:
+ 1, Unknown (MP3) (2.0 ch) (iso639-2: und)
+ 2, Unknown (AC3) (5.1 ch) (iso639-2: und), 48000Hz, 320000bps
+ subtitle tracks:
[17:10:44] 1 job(s) to process
[17:10:44] json job:
{
"Audio": {
"AudioList": [
{
"Encoder": 1125984256,
"Track": 0
}
],
"CopyMask": [
"copy:aac",
"copy:ac3",
"copy:eac3",
"copy:dtshd",
"copy:dts",
"copy:mp3",
"copy:truehd",
"copy:flac"
],
"FallbackEncoder": "fdk_aac"
},
"Destination": {
"ChapterList": [
{
"Name": ""
}
],
"ChapterMarkers": false,
"File": "out/bbb_sunflower_1080p_30fps_normal.m512.threads40_10.veryfast.1080p.out.mkv",
"Mp4Options": {
"IpodAtom": false,
"Mp4Optimize": false
},
"Mux": "mkv"
},
"Filters": {
"FilterList": [
{
"ID": 6,
"Settings": {
"mode": 1,
"rate": "27000000/900000"
}
},
{
"ID": 11,
"Settings": {
"crop-bottom": 0,
"crop-left": 0,
"crop-right": 0,
"crop-top": 0,
"height": 1080,
"width": 1920
}
}
]
},
"Metadata": {
"Artist": "Blender Foundation 2008, Janus Bager Kristensen 2013",
"Comment": "Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net",
"Composer": "Sacha Goedegebure",
"Genre": "Animation",
"Name": "Big Buck Bunny, Sunflower version"
},
"PAR": {
"Den": 1,
"Num": 1
},
"SequenceID": 0,
"Source": {
"Angle": 0,
"Path": "bbb_sunflower_1080p_30fps_normal.mp4",
"Range": {
"End": 1,
"Start": 1,
"Type": "chapter"
},
"Title": 1
},
"Subtitle": {
"Search": {
"Burn": true,
"Default": false,
"Enable": false,
"Forced": false
},
"SubtitleList": []
},
"Video": {
"ColorMatrixCode": 0,
"Encoder": "x264",
"Level": "4.1",
"OpenCL": false,
"Options": "threads=40:lookahead-threads=10",
"Preset": "veryfast",
"Profile": "high",
"QSV": {
"AsyncDepth": 4,
"Decode": false
},
"Quality": 18.5,
"Tune": "film",
"Turbo": false,
"TwoPass": false
}
}
[17:10:44] starting job
[17:10:44] Auto Passthru: allowed codecs are AAC, AC3, E-AC3, TrueHD, DTS, DTS-HD, MP3, FLAC
[17:10:44] Auto Passthru: fallback is AAC (FDK)
[17:10:44] Auto Passthru: using MP3 Passthru for track 1
[17:10:44] job configuration:
[17:10:44] * source
[17:10:44] + bbb_sunflower_1080p_30fps_normal.mp4
[17:10:44] + title 1, chapter(s) 1 to 1
[17:10:44] + container: mov,mp4,m4a,3gp,3g2,mj2
[17:10:44] + data rate: 3481 kbps
[17:10:44] * destination
[17:10:44] + out/bbb_sunflower_1080p_30fps_normal.m512.threads40_10.veryfast.1080p.out.mkv
[17:10:44] + container: Matroska (libavformat)
[17:10:44] * video track
[17:10:44] + decoder: h264
[17:10:44] + bitrate 2998 kbps
[17:10:44] + filters
[17:10:44] + Framerate Shaper (mode=1:rate=27000000/900000)
[17:10:44] + frame rate: 30.000 fps -> constant 30.000 fps
[17:10:44] + Crop and Scale (width=1920:height=1080:crop-top=0:crop-bottom=0:crop-left=0:crop-right=0)
[17:10:44] + source: 1920 * 1080, crop (0/0/0/0): 1920 * 1080, scale: 1920 * 1080
[17:10:44] + Output geometry
[17:10:44] + storage dimensions: 1920 x 1080
[17:10:44] + pixel aspect ratio: 1 : 1
[17:10:44] + display dimensions: 1920 x 1080
[17:10:44] + encoder: H.264 (libx264)
[17:10:44] + preset: veryfast
[17:10:44] + tune: film
[17:10:44] + options: threads=40:lookahead-threads=10
[17:10:44] + profile: high
[17:10:44] + level: 4.1
[17:10:44] + quality: 18.50 (RF)
[17:10:44] * audio track 1
[17:10:44] + decoder: Unknown (MP3) (2.0 ch) (track 1, id 0x1)
[17:10:44] + bitrate: 160 kbps, samplerate: 48000 Hz
[17:10:44] + MP3 Passthru
[17:10:44] sync: expecting 19038 video frames
[17:10:44] encx264: min-keyint: 30, keyint: 300
[17:10:44] encx264: encoding at constant RF 18.500000
[17:10:44] encx264: unparsed options: threads=40:lookahead-threads=10:ref=1:level=4.1:trellis=0:deblock=-1,-1:mixed-refs=0:weightp=1:subme=2:psy-rd=1,0.15:vbv-maxrate=62500:vbv-bufsize=78125:rc-lookahead=10
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
x264 [info]: profile High, level 4.1
[17:10:44] sync: first pts audio 0x1 is 0
[17:10:44] sync: first pts video is 6000
[17:10:44] sync: Chapter 1 at frame 1 time 6000
[17:13:04] reader: done. 1 scr changes
[17:13:05] work: average encoding speed for job is 135.756454 fps
[17:13:05] vfr: 19038 frames output, 0 dropped and 2 duped for CFR/PFR
[17:13:05] vfr: lost time: 0 (0 frames)
[17:13:05] vfr: gained time: 0 (0 frames) (0 not accounted for)
[17:13:05] mp3-decoder done: 26425 frames, 0 decoder errors
[17:13:05] h264-decoder done: 19036 frames, 0 decoder errors
[17:13:05] sync: got 19036 frames, 19038 expected
[17:13:05] sync: framerate min 30.000 fps, max 30.000 fps, avg 30.000 fps
x264 [info]: frame I:158 Avg QP:13.50 size:272210
x264 [info]: frame P:6559 Avg QP:17.86 size: 37082
x264 [info]: frame B:12321 Avg QP:21.33 size: 6672
x264 [info]: consecutive B-frames: 6.2% 17.9% 13.7% 62.2%
x264 [info]: mb I I16..4: 16.9% 23.3% 59.8%
x264 [info]: mb P I16..4: 3.9% 4.0% 1.0% P16..4: 26.4% 10.4% 6.5% 0.0% 0.0% skip:47.8%
x264 [info]: mb B I16..4: 0.5% 0.5% 0.1% B16..8: 9.5% 3.1% 0.6% direct: 3.8% skip:82.0% L0:36.7% L1:45.6% BI:17.7%
x264 [info]: 8x8 transform intra:41.3% inter:39.4%
x264 [info]: coded y,uvDC,uvAC intra: 51.9% 50.3% 22.2% inter: 8.4% 8.3% 1.0%
x264 [info]: i16 v,h,dc,p: 56% 23% 15% 7%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 20% 20% 4% 6% 10% 5% 5% 5%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 17% 12% 6% 8% 10% 7% 8% 7%
x264 [info]: i8c dc,h,v,p: 55% 20% 18% 7%
x264 [info]: Weighted P-Frames: Y:3.0% UV:1.5%
x264 [info]: kb/s:4644.59
[17:13:07] mux: track 0, 19038 frames, 368426106 bytes, 4644.27 kbps, fifo 1024
[17:13:07] mux: track 1, 26425 frames, 12684000 bytes, 159.89 kbps, fifo 2048
[17:13:07] libhb: work result = 0
Encode done!
HandBrake has exited.