muxer fixes for interleaving and mkv lockups

[van]

The patch at http://handbrake.fr/pastebin/pastebin.php?show=539 is a first cut at fixing the deadlock that causes lockups when the output container is mkv. It also attempts to give us better control of the way media streams are interleaved in mkv & mp4 containers in the hope it may cure some stuttering and latency problems. (The interleaving changes seem to work as intended for mkv but don't do much for mp4's because of a couple of problems in libmp4v2 described below.)

The basic idea is to make the muxer copy bufs from HB's fifos to internal per-track queues as quickly as possible. Since all the HB fifos are always emptied whenever muxer runs, there should be no possibility of deadlock. (To guarantee that the HB fifos can always be emptied, the internal queues expand as needed.)

Since we now have all the different media streams buffered in the muxer, we can output the streams interleaved in whatever size chunks we want. I chose 4 times the video frame rate, 100-125ms, since it looks like that's what Apple uses for HD (they seem to use 200-250ms for SD or maybe they changed defaults & HD is done with a newer encoder) and it makes sense in terms of startup latency & player internal buffer requirements.

The internal queues are output as soon as all of them have enough data to make up a chunk so the internal queue sizes are kept as small as they can be consistent with the temporal decoupling between the HB video & audio fifos (i.e., you have to hang onto audio frames until the video frames for the same time range show up & that can take a *long* time since video encoding is about 100 times slower than audio). I've temporarily changed the log level of the muxer track completion msg from 2 to 0 and added the internal fifo size to it so we can get an idea of the audio / video mismatch & maybe use it to retune (lower) the HB fifo sizes:

Code: Select all

mux: track 0, 2077 frames, 10861850 bytes, 1001.63 kbps, fifo 32
mux: track 1, 4049 frames, 1726678 bytes, 159.23 kbps, fifo 256
mux: track 2, 2699 frames, 4836608 bytes, 446.01 kbps, fifo 256

The problem with mp4 interleaving is that libmp4v2 has a 'default' chunksize of 1 second (which is *way* too large since it forces every player to buffer a second's worth of each media stream's data internally in addition to the decoder buffering plus adds almost two seconds to the startup latency when streaming). And although there's a MP4Track variable, m_durationPerChunk, that controls the chunking, I can't find any way to change it - it's a private member & there's no method or property tied to it. It would be nice if there were a way to set this. It would also be nice if the default were dropped to something in the 100-200ms range (but there are reasons why you want the interval to be tied to the frame rate & a fixed default isn't quite as good).

Anyway, it would be nice if people could give this a try & report any problems. Thanks.

[jbrjake]

Shiny =)

And although there's a MP4Track variable, m_durationPerChunk, that controls the chunking, I can't find any way to change it - it's a private member & there's no method or property tied to it. It would be nice if there were a way to set this. It would also be nice if the default were dropped to something in the 100-200ms range (but there are reasons why you want the interval to be tied to the frame rate & a fixed default isn't quite as good).

I pinged KonaBlend with a link to this thread, to expedite getting those mp4v2 changes made.

I've temporarily changed the log level of the muxer track completion msg from 2 to 0

Feel free to leave those messages at level 0 permanently. It was an oversight to hide them.

[JohnAStebbins]

I tried it out on a file that would have caused mkv to stall. Worked great.

[KonaBlend]

See changeset 2347 which uses new libmp4v2 w/ API to set m_durationPerChunk for both video/audio tracks to about 4 frames of video. I hope I made the correct assumption in setting the same duration for audio tracks.

[van]

Thanks KonaBlend - that works perfectly! I committed the muxer changes as r2348. It's amazing how much it changes the scrubbing behavior in QT player. With the old version it was very slow and jerky. With your new code it's instantaneous, seamless and very smooth.

I changed muxcommon so that that it always interleaves video and audio 1:1 so the container dependent muxer can rechunk as it sees fit (it's easy to chunk fully interleaved data). That means that chunking control for mp4s is entirely in muxmp4.c and containers like mkv that deal efficiently with 1:1 interleave will get it without any additional setup.

I also added muxer support for 'intermittent' tracks like subtitles or captions. We don't have support for these anywhere else in HB yet but if they get added at some point, muxcommon should deal with them.

jbrjake, as long as we're whacking on muxer stuff I'd like to change the video track sample rate in muxmp4.c to 90KHz and get rid of all the sample rate conversion cruft. For a lot of content this should reduce the size of the stts atom & just generally clean up the code & output file. I know that you, I and others have tested this. Any objections to changing it now or should we let the dust settle on the muxer changes first?

[jbrjake]

jbrjake, as long as we're whacking on muxer stuff I'd like to change the video track sample rate in muxmp4.c to 90KHz and get rid of all the sample rate conversion cruft.

Please do =)

[eddyg]

Hi,

What are other peoples experiences with MP4's and these changes on the AppleTV?

I've found that scrubbing does indeed appear slightly smoother - I didn't notice any significant pauses that were commonplace in the past. However I did notice pauses in normal play, for no apparent reason, the pause lasted about 3-5 seconds both audio and video, and then it sped up really fast to catch up to where it otherwise would have been.

I also tried in QuickTime and didn't notice much in the way of difference scrubbing wise, it certainly wasn't smooth.

I'm not certain that this was due to these changes, I need to do more troubleshooting, however I thought it worthwhile mentioning in case others have similar issues.

I'll continue to use this version for my EyeTV encodes, and hopefully will soon spot a pattern.

Cheers, Ed.

[dynaflash]

Am running some atv encodes now and will see how it goes.

[van]

I've found that scrubbing does indeed appear slightly smoother - I didn't notice any significant pauses that were commonplace in the past. However I did notice pauses in normal play, for no apparent reason, the pause lasted about 3-5 seconds both audio and video, and then it sped up really fast to catch up to where it otherwise would have been.

I also tried in QuickTime and didn't notice much in the way of difference scrubbing wise, it certainly wasn't smooth.

On the scrubbing I should have noted I was using the Universal preset (no bframes) since most of the encodes I do are for my daughter & need to play on either the TV or her touch. Scrubbing encodes with bframes is grim with or without the interleave changes. It gets reasonable if you set keyint_max to < twice the framerate so there are keyframes every second. But the encodes that result are bigger than what I get with Universal for the same content & quality.

The problems you're seeing might be something to do with splitting a P/B sequence across chunks. I tried a few DVDs and all the NZ clips from geekzone with the AppleTV preset and didn't see any misbehavior but the NZ broadcasters are always making weird changes in their bitstreams. What preset or settings are you using? Do you use CFR to limit the frame rate on 720p and, if so, are you using the patch from this thread?

[KonaBlend]

As per the following chart, stts atom table-entries is reduced significantly thanks to r2350 . I get the impression if we had no rounding issues stts would reduce to 1 as it does when (TIMESCALE % FPS == 0).

For 23.976 it oscillates between durations of 3753 and 3754 which is 1 frame in track timescale 90000.

Code: Select all

input_media  rev      stts
--------------------------
NTSC 23.98   svn2348  4626   
NTSC 23.98   svn2350  3085   

1080p24      svn2348     1  
1080p24      svn2350     1  

(x64 encodes were --quality 20 -x min-keyint=24:keyint=24 ).

[dynaflash]

Hi,

What are other peoples experiences with MP4's and these changes on the AppleTV?

I've found that scrubbing does indeed appear slightly smoother - I didn't notice any significant pauses that were commonplace in the past. However I did notice pauses in normal play, for no apparent reason, the pause lasted about 3-5 seconds both audio and video, and then it sped up really fast to catch up to where it otherwise would have been.

Well, after playing with two encodes I have not had any "lockups" during about four hours of viewing time on the AppleTV, plays smooth like butter.
Scrubbing does seem smoother, though it could be the placebo effect. Fwiw, scrubbing was always better on the atv with the eSata mod. Also, note, this is synced not streamed.

Settings: 62% vfr detelecine with:

Code: Select all

ref=3:mixed-refs=1:bframes=3:me=umh:subq=6:analyse=all:b-adapt=2:8x8dct=1

[cvk_b]

Code: Select all

--markers --previews=25 --verbose=2 -c 2-4 --audio=1,1,2 --aencoder=ac3,faac,ac3 --ab=auto,128,auto --mixdown=auto,stereo,auto --drc=0 --loosePixelratio=32 --width=704 --encoder=x264 --quality=20.4 --format=mkv -x cabac=0:ref=2:mixed-refs:me=umh:fast-pskip=0:stats=/Volumes/c2bin90mkv/cli.x264:pass=1

XBMC doing very strange things now @ 2351. No audio, video sped up, buffering.

Code: Select all

[08:50:07] mux: file size, 219363222 bytes
[08:50:07] mux: track 0, 21789 frames, 130842534 bytes, 1151.80 kbps, fifo 8
[08:50:07] mux: video bitrate error, +4728553103303901184 bytes
[08:50:07] mux: track 1, 28398 frames, 50889216 bytes, 447.98 kbps, fifo 128
[08:50:07] mux: track 2, 42598 frames, 14536699 bytes, 127.97 kbps, fifo 256
[08:50:07] mux: track 3, 28398 frames, 21809664 bytes, 191.99 kbps, fifo 128
[08:50:07] mux: overhead, 10.60 bytes per frame
[08:50:07] render: lost time: 0 (0 frames)
[08:50:07] render: gained time: 0 (0 frames) (0 not accounted for)

Will try other sources.