Chapters inside MP4 file

Cyander · Post by **Cyander** » Wed Mar 21, 2007 7:32 am

Chapter tracks aren't as ugly as it first appears.

I have been trying to figure this out myself, and it is kinda frustrating because I am so close, yet so far from the goal.

For the .m4v files iTMS sells, the chapter tracks are MPEG-4 Timed Text tracks. You can actually extract the raw track using mp4box, and I am trying to see exactly how close to the MPEG-4 Timed Text track spec Apple's tracks follow.

The problem is that mp4box imports Text Tracks using a subtype that Apple doesn't recognize in Quicktime, even though the data looks (almost) identical. Worse still, is that since libmp4v2 has no code to support MPEG-4 Timed Text tracks, support in Handbrake also means writing/submitting a patch to the MPEG4IP team for decent Timed Text track support.

EDIT: I just broke through the nasty part... I successfully dumped a chapter track from a purchased iTMS movie using mp4box. Here is the great news: It is just a standard MPEG-4 Timed Text Track. Nothing shifty here at all other than the 'different' subtype of "text" as the fourcc instead of "3gtx" that mp4box forces.

That said, it still means that libmp4v2 needs to be updated to support MPEG-4 Timed Text Tracks before Handbrake can add chapters to MP4 files. And someone still needs to figure out how to connect a sound track to a chapter track... which atom needs to be set/added, and where. I suppose I could look at it if I have the time.

A 'good' side-effect of knowing how to do chapter tracks, is that text track subtitles are also possible using the same method (once you get around the nastiness that is character recognition).

maurj · Post by **maurj** » Wed Mar 21, 2007 10:55 am

Hi Cyander,

I've just been looking at a iTunes Store movie here, and it seems to have two tracks for chapters; the Timed Text track for the chapter names and times, and also a JPEG video track with still images for each chapter. If you watch one of these movies in iTunes, then the "separate window" view has a chapter popup menu in its controller which uses these jpegs, together with the text and timings, to make a nice chapter display menu.

It should in theory be possible to do all of this for Handbrake's movies - but as you say, the trick is getting libmp4v2 to create these tracks.

I think libmp4v2 is reasonably well set up to be extensible to create one's own atoms, so it's probably just a case of putting in the time with Dumpster and a hex editor to find out what's expected by iTunes / Front Row.

Likewise, if I get time, I might investigate this further!

Cyander · Post by **Cyander** » Wed Mar 21, 2007 3:39 pm

Interesting... iTunes didn't actually have the menu in 7.0.2, so it looks like they are turning on the iTunes support for chapter tracks.

I noticed the photo track, and even mucked with it, coming to the same conclusion, but not seeing where Apple was planning on enabling it. Guess my question is now answered.

I don't see the photo track as a priority though, as iTunes will gladly make chapters available for movies that don't have it. I took my copy of Indiana Jones and the Last Crusade, and added an old text track I made in an attempt to add chapters earlier, dumping it into a .mov container.

iTunes happily played it back, and allowed me to access the chapters, so it definitely isn't a requirement that HB support the photo track.

Which means, that someone can likely just visit the MPEG-4 specs on how timed tracks work (or look at the GPAC library that mp4box uses), incorporate the functionality into libmp4v2 since that is where the functionality should go, and then use it in Handbrake with a UI to let the user name the chapters they are ripping in a table or something. Then figure out which atom is used for telling a video/sound track it has a chapter track.

EDIT: Aha! Caught ya!

Code: Select all

Atom moov @ 32 of size: 3065984, ends @ 3066016
    Atom trak @ 148 of size: 1443934, ends @ 1444082
         Atom tref @ 284 of size: 24, ends @ 308
             Atom chap @ 292 of size: 16, ends @ 308

As I started to suspect, the atom is just a simple Track Reference atom, right out of the QT specs. Quicktime uses this to reference one track from another, with the 'chap' atom being used specifically to denote a chapter track. This in particular is from "Clue", and the track in question is the audio track.

It gets better, I know how the atom works, so we can use libmp4v2 to write this atom.

Code: Select all

------ tref Atom ------
4B Length
4B Atom Code - 'tref'
    ------ chap Atom ------
    4B Length
    4B Atom Code - 'chap'
    4B (Repeating) Track IDs
    --------------------------
------------------------

The length and atom code is handled by libmp4v2 IIRC, but I included the full structure of the Track Reference atom for completeness.

So, in the case of an iTunes movie, the chap atom has a length of 0x10 (16 bytes). 8 Bytes of header, 8 bytes of track IDs. The Track IDs I saw in clue were: 0x00000004 and 0x00000005. Track 4 was the text track, and track 5 was the photo track.

Do I get a cookie?

maurj · Post by **maurj** » Wed Mar 21, 2007 5:04 pm

You most certainly do!

At much the same time, I've been doing some munging around inside an iTunes Store movie file, to find out how it stores its chapter info. The text and jpeg tracks are both in there, and I've kind of sussed out how they are working. The audio track has a 'chap' tref pointing to the text track and the jpeg track. The text track (and likewise the jpeg track) have their data split throughout the video media stream. I guess so this is they are encountered when needed in the stream, rather than being tacked on the end.

I've been playing around with getting libmp4v2 to include new atom types to do this with some success, and have laid the groundwork for adding a 'text' track (it doesn't know about text tracks atm). This will be the trickier bit (the 'chap' atom is easy to add). Good find though - this should give what I need to link it all together!

Kurtz · Post by **Kurtz** » Wed Mar 21, 2007 6:26 pm

Good work guys
Never would have found that information out!

Does MF/HB have a way of reading dvd ifo info for chapter markers. I found a couple of scripts that do this but thats all.

Keep the exceptional work up and we might have a seriously cool feature.

Cyander · Post by **Cyander** » Wed Mar 21, 2007 6:36 pm

Kurtz wrote:Does MF/HB have a way of reading dvd ifo info for chapter markers. I found a couple of scripts that do this but thats all.

Handbrake already reads the information for chapter breaks, but I am not familiar with the format enough to say if information like the chapter title is encoded on the DVD itself or not. Seeing what other solutions exist, am I leading towards not. But, as we already read the chapter breaks, it isn't brain surgery to use that for the chapter timecodes (with some logic to shift the timecodes when only encoding some of the chapters).

Maurj wrote: At much the same time, I've been doing some munging around inside an iTunes Store movie file, to find out how it stores its chapter info. The text and jpeg tracks are both in there, and I've kind of sussed out how they are working. The audio track has a 'chap' tref pointing to the text track and the jpeg track. The text track (and likewise the jpeg track) have their data split throughout the video media stream. I guess so this is they are encountered when needed in the stream, rather than being tacked on the end.

The text track is a standard MPEG-4 timed text track, as per the ISO spec. If you want to see code that already understands this track type, look at mp4box/GPAC. As that is what I used to dump the text track from "Clue". Timecodes and everything match up, so there isn't anything Apple-special about it other than the subtype of 'text'.

As you have already discovered, timed text tracks are really quite nice, since you can break up the samples and insert them at the right timecodes when muxing.

I haven't even looked to see what specs exist regarding the JPEG track Apple is using, so I don't know if it deviates from the ISO spec, and if so, by how much.

So far though, the only 'non-standard' use of the MP4 format is the tref atom, and that is still a pretty standard QT atom.

Kurtz · Post by **Kurtz** » Wed Mar 21, 2007 6:50 pm

Obviously it reads the chapter times for chapter selections.
Thanks for putting me straight Cyander.

Is this being looked at in the Dev forums or just here? cant find any reference to it, but wandered if there more people to pool resources.

Cyander · Post by **Cyander** » Wed Mar 21, 2007 8:17 pm

Kurtz wrote:Obviously it reads the chapter times for chapter selections.
Thanks for putting me straight Cyander.

Is this being looked at in the Dev forums or just here? cant find any reference to it, but wandered if there more people to pool resources.

Since I am already working on some improvements to the subtitle burn-in process, and this is a feature I would definitely like to see, if nobody else is going to work on it, I will.

If someone is already working on it, I can throw together a patch or two for parts of it. I have a pretty good feel for what I want the UI to look like, and how to wire it up from the front-end, as well as the tref/chap atom.

jbrjake · Post by **jbrjake** » Wed Mar 21, 2007 8:30 pm

I'm just watching quietly, hoping for Cynander's hard work to pay off in a patch.

So I ain't touching nothing, not even moving the thread to the Dev forum. I don't want to ruin it =)

maurj · Post by **maurj** » Wed Mar 21, 2007 10:59 pm

Hi Cyander,

I've been working on patching mp4v2 today, to add 'text' track support... I'm a good way there, but will be away for a few days from tomorrow, so if you want to try it separately, do give it a go! Otherwise, I'll persevere when I'm back - think I know everything I need to do now

Cyander · Post by **Cyander** » Wed Mar 21, 2007 11:11 pm

I am likely spending the next couple of days on subtitle work, and probably the 'tref' atom work since I have code stuck in my brain for that.

maurj · Post by **maurj** » Wed Mar 21, 2007 11:25 pm

Hi Cyander,

Good stuff! I'll keep going with the chapter / text track stuff, then.

Btw, mp4v2 has code to manage track references already (albeit not 'chap' chapter ones, though they're easily added). Take a look at the AddTrackReference function (and similar) in mp4file.cpp. I've added 'chap' to mp4v2 as part of the work I'm doing.

Cyander · Post by **Cyander** » Thu Mar 22, 2007 12:02 am

maurj wrote:Btw, mp4v2 has code to manage track references already (albeit not 'chap' chapter ones, though they're easily added). Take a look at the AddTrackReference function (and similar) in mp4file.cpp. I've added 'chap' to mp4v2 as part of the work I'm doing.

Nevermind then.

Is there a branch that you will be checking this stuff into before you head out for a couple days? I might be interested in moving it forward a bit while you are gone if the subtitle work is simple.

jbrjake · Post by **jbrjake** » Thu Mar 22, 2007 12:30 am

Cyander wrote:Is there a branch that you will be checking this stuff into before you head out for a couple days? I might be interested in moving it forward a bit while you are gone if the subtitle work is simple.

maurj's already had one branch for this release cycle and that's more than anyone else ;P

More seriously, there's been a vague sort of consensus to chill out on branches for awhile. At least til 0.8.5b1 is done. Everyone's been committing right to the trunk the past few weeks and it's working out ok. Makes it a lot easier to follow changesets on the Trac, too.

maurj · Post by **maurj** » Thu Mar 22, 2007 1:05 am

Heh!

Cyander, I'll try and get this working and commit it in the next few days. I really don't want to hold up your enthusiasm or hold you back. Since the changes I'm making involve a fairly major patch to mp4v2, it's not quite as simple as me checking in the changes, but I'll try and share it as and when there's something working.

At the moment, I have mp4v2 creating all of the new atoms needed for chapter text tracks, but for some reason QuickTime Pro isn't recognising the tracks. Probably some small but crucial flag setting. I'll keep persevering and will get there in the end...

Cyander · Post by **Cyander** » Thu Mar 22, 2007 6:01 am

Well, that is why I asked if this was going to get commited to a branch. Usually easier to muck around with unstable changes and share them that way without destabilizing the trunk too much.

One thing I noticed with mp4box is that it creates text tracks slightly differently, and the sub-type isn't compatible...

maurj · Post by **maurj** » Fri Mar 23, 2007 11:33 am

Aha! I have it working!

Well, specifically I can now create chapter text tracks in movies, and write text samples to them. I've not quite got the text formatting right, and I'm not writing the actual DVD chapter times yet, but I can add chapter tracks with the right track reference and write chapters to them.

There's only one slight wrinkle. I spent a lot of time trying to work out why QuickTime wouldn't recognise my chapter tracks in an mp4 file. It seems that it simply won't even acknowledge a text track in a file with an .mp4 extension. This is fair enough - text tracks aren't part of the mp4 specification.

Problem is, if you rename a Handbrake mp4 file to have an extension of mov, then QuickTime won't read it either for some reason (even though strictly speaking an mp4 is a mov).

But, I've found a workaround! If you call the exact same file "something.m4v", then the chapter track works! Quicktime, iTunes and Front Row will recognise the chapter track and display it. it's as if m4v is some hybrid between mov and mp4.

I'll work on setting the actual DVD chapter times in the track. This raises a couple of questions:

How keen are we to make the chapter track be a default for the mp4 format (given that it stops it being a strict mp4)?

If this should be a default option, would we be happy to change HandBrake to create files with an m4v extension, rather than mp4? What would this mean for non-QuickTime playback / other platforms such as linux?

Should chapter track inclusion be a prefs option, which forces the m4v extension (rather than mp4) if enabled?

Should we let people enter their own chapter names? (I'm using "Chapter 1", "Chapter 2" etc. for now).

Thoughts much appreciated!

maurj.

maurj · Post by **maurj** » Fri Mar 23, 2007 12:30 pm

Extra info: got it all working now, using the real DVD chapter times as discovered by Handbrake. Lovely!

rhester · Post by **rhester** » Fri Mar 23, 2007 2:18 pm

How similar is your solution to the one offered by MP4Box? Can the MP4Box chapter data be "fixed" to work with iTunes? If not, would it be possible for us to additionally create a small standalone app to add chapters to pre-existing rips?

Rodney

tanalach · Post by **tanalach** » Fri Mar 23, 2007 2:53 pm

maurj,

Great work... thanks for the effort! You're definitely getting ahead of the bazillions of AppleTV folks who are going to show up shortly asking for chapter marks.

Re: your questions:

- Chapter marks should be an option, not default
- Selection of chapter marks should force an m4v file extension
- Non-selection should default to mp4
- Override of file extension would be nice in the mp4 case (i.e. make m4v without chapters is okay, not the other way around

Thanks again,
T

Kurtz · Post by **Kurtz** » Fri Mar 23, 2007 4:40 pm

That was really fast! good work maurj.

In answer to your questions i aggree with tanalach set up

As for chapter names, I think being able to name them would be cool but I dont see myself typing out 20 or so names just so i can see them in itunes or quicktime (they dont show up in front row), so not essential. so defaults in my opinion would be the way to go.

Of course having the option is always nice.

Again thanks for the work you have put in.

tanalach · Post by **tanalach** » Fri Mar 23, 2007 7:23 pm

TEST REPORT!

I grabbed maurj's code and did a quick test. Outstanding results!

The chapter marks were preserved, and both iTunes and AppleTV played them without a single problem. Seek time on the AppleTV (streaming) was a couple of seconds. I haven't tried sync mode yet but if it's anything like purchased media it'll be instant.

Great, great work!

(PS the chapter code is only in cli now, so I didn't use the GUI)

Cyander · Post by **Cyander** » Fri Mar 23, 2007 7:38 pm

You definitely beat me, congrats!

I'll take a look at the code and if it hasn't already been done, do the work to hook it up in the OS X UI tonight.

dynaflash · Post by **dynaflash** » Fri Mar 23, 2007 7:45 pm

tanalach wrote:(PS the chapter code is only in cli now, so I didn't use the GUI)

I just checked in the gui chapter markers to the svn. its rev 441!
Have fun!

Cyander · Post by **Cyander** » Fri Mar 23, 2007 7:48 pm

/cry

Oh well, I still have subtitle work to do anyways.

HandBrake

Chapters inside MP4 file

Good work

Re: Good work

DVD chapters

Re: DVD chapters

That was quick!