a/v sync breakage in mkv at svn1542

Archive of historical development discussions
Discussions / Development has moved to GitHub
Forum rules
*******************************
Please be aware we are now using GitHub for issue tracking and feature requests.
- This section of the forum is now closed to new topics.

*******************************
Post Reply
User avatar
JohnAStebbins
HandBrake Team
Posts: 5583
Joined: Sat Feb 09, 2008 7:21 pm

a/v sync breakage in mkv at svn1542

Post by JohnAStebbins »

Van, fyi, something you did with the buffering.

Log: http://handbrake.pastebin.ca/1065023
van
Veteran User
Posts: 417
Joined: Wed Aug 29, 2007 6:35 am

Re: a/v sync breakage in mkv at svn1542

Post by van »

Sorry John. The problem should be fixed by r1559.

R1542 was a fix for occasional HB hangs that turned out to be due to a deadlock between reader & mux. The HB processing pipeline looks like this:
Image
The input file is demuxed by Reader into separate video & audio pipelines that are decoded, time aligned, cleaned up & re-encoded independently then merged by "Mux" into the output container. Whenever you have a flow diagram with two or more constrictions (Reader & Mux in this case) there's a possibility of deadlock. The typical lockup I saw was the bottom branch getting filled with bufs because of silence being generated to fill a gap. So Reader was blocked waiting to inject a new audio buf into a full pipe. Meanwhile Mux had emptied all the video out of the top branch & was waiting for video because it required that a buf be available on every track so that it could deliver them in strict DTS order to the container-specific mux routine. Classic deadlock: Reader waiting for Mux to consume an audio while Mux waiting for Reader to produce a video.

R1542 removed the coupling between audio & video in mux by not requiring that all tracks have a buf. Instead it would take the track with the earliest buf but ignored tracks with no buf. This still guarantees that each track will get its bufs in strict DTS order but does allow one track to get ahead of another by up to the pipeline FIFO depth (around 2000 bufs worst case which is 60 seconds for 30FPS NTSC video or AC3 audio). This is no problem for mp4 or avi containers since they have no cross-track dependencies. It wasn't a problem with the mkv test case I ran but it was a problem in general because I didn't understand that mkv chunks interleave data from different tracks & mkv readers (quite reasonably) assume that audio & video that are close in time will be in the same chunk or chunks that are close. With r1542 video could easily end up in a chunk with audio from ten seconds earlier which drove VLC bonkers.

R1559 restores the old mux behavior for mkv containers (the new mux behavior is still used for mp4 & avi). This fix is a stopgap while I come up with something better. I want the new behavior to work with all containers since:

- it can't deadlock

- it gracefully tolerates video without audio (or vice-versa) so we can rid of the current silliness where a bunch of extra silence is generated at the end of the video content to make sure the muxer has something in the audio fifos so it will be able to consume all the video content (you see this as "generating silence" messages in the log). And we can finally stop ignoring DVD titles that have video but no audio.

- it improves parallelism since mux no longer blocks if one of the fifos goes empty. This seems to generate substantial peformance improvement on multicore platforms with poor thread schedulers (ie., Windows). Since Mux used to sleep for 50ms if any fifo went empty, you could end up with encoder threads being blocked for long periods waiting for Mux to clean stuff out of their fifo while Mux was sleeping waiting for some other encoder to put something in its fifo and the other encoder wasn't getting the cycles it needed to do this because the Windows scheduler was giving them to the first (blocked) encoder since it hadn't exhausted its quantum yet.
saintdev
Regular User
Posts: 146
Joined: Wed Dec 20, 2006 4:17 am

Re: a/v sync breakage in mkv at svn1542

Post by saintdev »

Thanks for posting that van. I was wondering what you meant by the comment in r1559. I eventually figured it out, but this helps clarify what's going on. Also, from the same comment we weren't sure that you realized that libmkv is our (well my) own creation, and not something from the matroska team. So, if there's any suggestions you have, I'd be glad to listen.
Although this really seems to be a player bug. That is, the player (VLC especially, but their matroska demuxer is crap anyway ;) ) is assuming that blocks from different tracks with the same timecode will be located physically nearby to each other. MPlayer seems to handle this well; audio and video get out of sync when you seek, but slowly converge again. I'll have to test, but this could possibly be fixed by writing explicit CuePoints (mapping between timecode and file-position) for each track, instead of just for video as it is now. But, not every player honors CuePoints, so this may not work.
It does also bring up a bug in libmkv that I'm not entirely sure how to fix. That is mainly it ends up writing waay too many Clusters. It's currently setup to write a new Cluster if the new timecode is 2s > or < the one from the current cluster. After a few hundred seconds of video this results in a new Cluster being written every time HandBrake switches tracks. This is a _lot_ of unnecessary overhead. I'm thinking about switching to just writing new Clusters based on how many MB of the file we have written (this is already taken into account, but I was just thinking of removing the timecode check).
Post Reply