Turbo 1st pass slower than 2nd pass

[AgentMax]

Hi,

I'm using the latest SVN 2907 x64 on OSX10.6.2 with a Core i7 3,06GHz. 4 cores+4 HT cores.

Encoding a mkv (or DVD doesn't matter, only a bit faster) with the following settings:

Code: Select all

level=30:cabac=0:ref=3:mixed-refs=1:bframes=6:direct=auto:no-fast-pskip=1:me=umh:subq=7:b-adapt=2:analyse=all:8x8dct=0

Bitrate average 5000. 2 pass. AAC 128. AC3 Pass thru.

As you can see in the activity log, the turbo first past is slower (24fps) than the "real" 2nd pass (34fps).

Turbo first pass is using around 200% CPU.
2nd pass is using 800% CPU.

There's something wrong, isn't it?

Thanks, Michael.

Code: Select all

[16:15:20] macgui: performNewQueueScan currentQueueEncodeIndex is: 65
[16:15:20] hb_scan: path=/Users/Michael/Downloads/Downloads/knri-hddl-s01e07.mkv, title_index=1
[16:15:20] scan: trying to open with libdvdread
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Using libdvdcss version 1.2.10 for DVD access
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[16:15:20] dvd: not a dvd - trying as a stream/file instead
Input #0, matroska, from '/Users/Michael/Downloads/Downloads/knri-hddl-s01e07.mkv':
  Duration: 00:40:01.28, start: 0.000000, bitrate: N/A
    Stream #0.0: Video: h264, yuv420p, 1280x720, PAR 1:1 DAR 16:9, 25 tbr, 1k tbn, 50 tbc
    Stream #0.1: Audio: ac3, 48000 Hz, stereo, s16
    Stream #0.2: Audio: ac3, 48000 Hz, 5.1, s16
[16:15:21] scan: decoding previews for title 1
[16:15:21] scan: audio 0x1: AC-3, rate=48000Hz, bitrate=192000 Unknown (AC3) (2.0 ch)
[16:15:21] scan: audio 0x2: AC-3, rate=48000Hz, bitrate=384000 Unknown (AC3) (5.1 ch)
[16:15:25] scan: 10 previews, 1280x720, 25.000 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 1:1
[16:15:25] scan: title (0) job->width:1280, job->height:720
[16:15:25] libhb: scan thread found 1 valid title(s)
[16:15:25] macgui: ScanDone state received from fQueueEncodeLibhb
[16:15:25] macgui: Preset: ATV HD
[16:15:25] macgui: processNewQueueEncode number of passes expected is: 2
[16:15:25] macgui: prepareJob exiting
[16:15:26] 2 job(s) to process
[16:15:26] starting job
[16:15:26] job configuration:
[16:15:26]  * source
[16:15:26]    + /Users/Michael/Downloads/Downloads/knri-hddl-s01e07.mkv
[16:15:26]    + title 1, chapter(s) 1 to 1
[16:15:26]    + container: matroska
[16:15:26]  * destination
[16:15:26]    + /Users/Michael/Desktop/knri-hddl-s01e07.m4v
[16:15:26]    + container: MPEG-4 (.mp4 and .m4v)
[16:15:26]  * video track
[16:15:26]    + decoder: h264
[16:15:26]    + frame rate: same as source (around 25.000 fps)
[16:15:26]    + dimensions: 1280 * 720 -> 1280 * 720, crop 0/0/0/0
[16:15:26]    + encoder: x264
[16:15:26]      + options: level=30:cabac=0:ref=3:mixed-refs=1:bframes=6:direct=auto:no-fast-pskip=1:me=umh:subq=7:b-adapt=2:analyse=all:8x8dct=0:ref=1:subme=1:me=dia:analyse=none:trellis=0:no-fast-pskip=0:8x8dct=0:weightb=0
[16:15:26]      + bitrate: 5000 kbps, pass: 1
[16:15:26]  * audio track 0
[16:15:26]    + decoder: Unknown (AC3) (2.0 ch) (track 1, id 1)
[16:15:26]      + bitrate: 192 kbps, samplerate: 48000 Hz
[16:15:26]    + mixdown: Stereo
[16:15:26]    + encoder: ca_aac
[16:15:26]      + bitrate: 128 kbps, samplerate: 48000 Hz
libdvdnav: Using dvdnav version 4.1.3
x264 [warning]: frame MB size (80x45) > level limit (1620)
x264 [warning]: MB rate (90000) > level limit (40500)
libdvdread: Using libdvdcss version 1.2.10 for DVD access
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[16:15:26] dvd: not a dvd - trying as a stream/file instead
[16:15:26] reader: first SCR 0 id 0 DTS 0
x264 [info]: profile Main, level 3.0
No accelerated IMDCT transform found
[16:15:26] sync: expecting 60057 video frames
[16:57:52] reader: done. 1 scr changes
[16:58:06] sync: got 60031 frames, 60057 expected
[16:58:06] work: average encoding speed for job is 23.446310 fps
[16:58:10] h264-decoder done: 60032 frames, 0 decoder errors, 0 drops
[16:58:10] render: lost time: 0 (0 frames)
[16:58:10] render: gained time: 0 (0 frames) (0 not accounted for)
x264 [info]: frame I:738   Avg QP:13.33  size:110062  PSNR Mean Y:50.80 U:55.46 V:56.35 Avg:51.91 Global:51.55
x264 [info]: frame P:27239 Avg QP:16.53  size: 36889  PSNR Mean Y:47.28 U:51.56 V:52.28 Avg:48.32 Global:47.84
x264 [info]: frame B:32054 Avg QP:18.75  size: 12625  PSNR Mean Y:46.54 U:51.20 V:51.99 Avg:47.64 Global:47.18
x264 [info]: consecutive B-frames:  6.7% 60.2% 11.9% 19.3%  1.2%  0.6%  0.1%
x264 [info]: mb I  I16..4: 18.4%  0.0% 81.6%
x264 [info]: mb P  I16..4: 24.5%  0.0%  0.0%  P16..4: 67.8%  0.0%  0.0%  0.0%  0.0%    skip: 7.7%
x264 [info]: mb B  I16..4:  6.8%  0.0%  0.0%  B16..8: 36.9%  0.0%  0.0%  direct:19.2%  skip:37.0%  L0:36.1% L1:46.0% BI:17.9%
x264 [info]: final ratefactor: 16.62
x264 [info]: direct mvs  spatial:99.6% temporal:0.4%
x264 [info]: coded y,uvDC,uvAC intra: 54.5% 63.6% 27.9% inter: 27.0% 29.4% 1.1%
x264 [info]: i16 v,h,dc,p: 30% 26% 22% 22%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 22% 14%  8%  7%  7%  7%  6%  8%
x264 [info]: SSIM Mean Y:0.9890743
x264 [info]: PSNR Mean Y:46.929 U:51.411 V:52.173 Avg:48.000 Global:47.506 kb/s:4966.49
[16:58:11] starting job
[16:58:11] job configuration:
[16:58:11]  * source
[16:58:11]    + /Users/Michael/Downloads/Downloads/knri-hddl-s01e07.mkv
[16:58:11]    + title 1, chapter(s) 1 to 1
[16:58:11]    + container: matroska
[16:58:11]  * destination
[16:58:11]    + /Users/Michael/Desktop/knri-hddl-s01e07.m4v
[16:58:11]    + container: MPEG-4 (.mp4 and .m4v)
[16:58:11]  * video track
[16:58:11]    + decoder: h264
[16:58:11]    + frame rate: same as source (around 25.000 fps)
[16:58:11]    + dimensions: 1280 * 720 -> 1280 * 720, crop 0/0/0/0
[16:58:11]    + encoder: x264
[16:58:11]      + options: level=30:cabac=0:ref=3:mixed-refs=1:bframes=6:direct=auto:no-fast-pskip=1:me=umh:subq=7:b-adapt=2:analyse=all:8x8dct=0
[16:58:11]      + bitrate: 5000 kbps, pass: 2
[16:58:11]  * audio track 0
[16:58:11]    + decoder: Unknown (AC3) (2.0 ch) (track 1, id 1)
[16:58:11]      + bitrate: 192 kbps, samplerate: 48000 Hz
[16:58:11]    + mixdown: Stereo
[16:58:11]    + encoder: ca_aac
[16:58:11]      + bitrate: 128 kbps, samplerate: 48000 Hz
libdvdnav: Using dvdnav version 4.1.3
x264 [warning]: frame MB size (80x45) > level limit (1620)
x264 [warning]: DPB size (3 frames, 4147200 bytes) > level limit (2 frames, 3110400 bytes)
x264 [warning]: MB rate (90000) > level limit (40500)
libdvdread: Using libdvdcss version 1.2.10 for DVD access
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.IFO failed
libdvdnav:DVDOpenFileUDF:UDFFindFile /VIDEO_TS/VIDEO_TS.BUP failed
libdvdread: Can't open file VIDEO_TS.IFO.
libdvdnav: vm: failed to read VIDEO_TS.IFO
[16:58:11] dvd: not a dvd - trying as a stream/file instead
[16:58:11] reader: first SCR 0 id 0 DTS 0
x264 [info]: profile Main, level 3.0
No accelerated IMDCT transform found
[16:58:12] sync: expecting 60031 video frames
[17:28:02] reader: done. 1 scr changes
[17:28:11] sync: got 60031 frames, 60031 expected
[17:28:11] work: average encoding speed for job is 33.363548 fps
[17:28:14] mux: track 0, 60031 frames, 1500670082 bytes, 4999.65 kbps, fifo 64
[17:28:14] mux: track 1, 112473 frames, 39525486 bytes, 131.68 kbps, fifo 1024
[17:28:14] h264-decoder done: 60032 frames, 0 decoder errors, 0 drops
[17:28:14] render: lost time: 0 (0 frames)
[17:28:14] render: gained time: 0 (0 frames) (0 not accounted for)
x264 [info]: frame I:738   Avg QP:13.84  size:105945  PSNR Mean Y:50.66 U:56.50 V:57.37 Avg:51.91 Global:51.60
x264 [info]: frame P:27239 Avg QP:16.03  size: 36632  PSNR Mean Y:48.26 U:53.04 V:53.86 Avg:49.38 Global:49.03
x264 [info]: frame B:32054 Avg QP:18.17  size: 13249  PSNR Mean Y:47.61 U:52.45 V:53.33 Avg:48.75 Global:48.45
x264 [info]: consecutive B-frames:  6.7% 60.2% 11.9% 19.3%  1.2%  0.6%  0.1%
x264 [info]: mb I  I16..4: 15.7%  0.0% 84.3%
x264 [info]: mb P  I16..4:  6.3%  0.0% 12.4%  P16..4: 43.6% 22.9%  8.3%  0.5%  0.4%    skip: 5.7%
x264 [info]: mb B  I16..4:  3.7%  0.0%  2.6%  B16..8: 43.7%  2.3%  1.9%  direct:10.5%  skip:35.4%  L0:41.6% L1:46.2% BI:12.2%
x264 [info]: direct mvs  spatial:93.7% temporal:6.3%
x264 [info]: coded y,uvDC,uvAC intra: 63.4% 75.8% 39.1% inter: 22.3% 26.1% 1.4%
x264 [info]: i16 v,h,dc,p: 30% 23% 20% 27%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 23% 19%  5%  7%  7%  7%  5%  5%
x264 [info]: ref P L0: 72.5% 18.2%  9.3%
x264 [info]: ref B L0: 80.2% 19.8%
x264 [info]: SSIM Mean Y:0.9908807
x264 [info]: PSNR Mean Y:47.945 U:52.765 V:53.620 Avg:49.073 Global:48.731 kb/s:4999.75

[AgentMax]

With version 0.9.4 it got worse.

Only 150% CPU load (800% available) and only 18 fps average...

[s55]

There is nothing wrong. This is to be expected. First pass won't always use an 8 Core machine.

[AgentMax]

Hm, why not?

[Deleted User 11865]

Hm, why not?

The faster "turbo" options aren't necessarily as well threaded as the normal options.

Also, as x264 developer Dark_Shikari pointed out on IRC the other day, mb-tree lookahaead (and, IIRC, b-adapt 2) are only performed during the first pass, so that can slow down the first pass quite a bit too (turbo or not).

One thing you could try is using normal first pass instead of turbo and see if it improves performance.

[AgentMax]

Sure, but why is the encoding process not using all cores on 1st pass.
So it would be faster. Are there any limitations?
I don't understand WHY it's not using everything it could for better performance.

As i posted above with 0.9.4
First Pass Turbo 18 fps (150% CPU)
Second Pass 26 fps (800% CPU)

I would like to hear something like "1st pass is slower because..... and it have to be so slow" or something like that.
I don't understand the technical limitation for a faster 1st pass.

[Deleted User 11865]

The faster "turbo" options aren't necessarily as well threaded as the normal options.

Put another way, x264 developers put more time into optimizing the slower options to scale better on multi-core machines. The turbo first pass uses some faster options that apparently don't scale very well. They're still faster on most machines (with 2 or less cores), but in your case they're not.

What happens if you do a 2-pass encode but uncheck turbo first pass, can you post a log for that?

[baggss]

I would like to hear something like "1st pass is slower because..... and it have to be so slow" or something like that.
I don't understand the technical limitation for a faster 1st pass.

The answer is: That's how it was designed/coded.