QT Subtitles & Closed Captioning (Mostly Bad News)

Archive of historical development discussions
Discussions / Development has moved to GitHub
Forum rules
*******************************
Please be aware we are now using GitHub for issue tracking and feature requests.
- This section of the forum is now closed to new topics.

*******************************
Post Reply
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

QT Subtitles & Closed Captioning (Mostly Bad News)

Post by Cyander »

I spent some time investigating the recent subtitles and closed captioning support available in iTunes and on the iPhone/iPod Touch.

First, the good news... closed captioning as it is used in iTunes is already documented. It is just a clcp/c608 track (Quicktime Closed Captioning) in an MPEG-4 file. There is even a sample on how to convert Scenarist CC files and embed them into a Quicktime file. Supporting CC in Handbrake is probably quite possible (since you can also convert DVD CC data into a Scenarist CC file easily, but I am not aware if code is available for that).

Now, the semi-bad news... subtitles in Quicktime appear to be implemented as a form of MPEG-4 Timed Text tracks. They even use the text/tx3g track type from what I can tell. The downside here is that the old Mac build of mp4box will create a file that Quicktime won't open (invalid public atom), which before Quicktime would just ignore the text track.

The worst news is that there seems to be no support for subpicture-style subtitles (the way Nero does it, or would be possible using Handbrake). :(
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by jbrjake »

Seems to me there are three choices: using the CCs, OCR, or just letting people import text subs found in the wild.

CCs: Bad choice for people without hearing difficulties, although it would be nice to have as an option for those who do. Example...when the 2nd Ghost in the Shell movie was released on DVD in the US, they accidentally included the CC instead of the sub track. So you'd hear a helicopter whirring, and on screen would be: "[The sound of a helicopter comes from above]". It was a big enough problem that the studio had to replace the discs.

OCR: Bad choice for people without a lot of spare time to correct the OCR engine's bad guesses.

Importing: Complicated for users. Hard for us to recommend sites that offer them. And it's kind of a crass process...most of the sites are full of ugly flashing banners and pop-ups. Plus, isn't it kind of...shady? However, importing existing text subs found in the wild is probably the best choice, imo.

Unless...can we turn the bitmap vob subs into ascii art? ;p
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by Cyander »

Yeah, I have the second movie, and know what you mean.

However, CC and Subtitles in this case are two different things (even to Apple). CC is very limited compared to subtitles, and still uses the fairly old system that has been in use with TV broadcasts for quite awhile. AFAIK, it still uses ASCII + control codes, so no international support there. Plus, regardless of subtitle support, CC support could be useful. CC on DVD is very similar to CC in the SCC format, which is very similar to CC in Quicktime. For the most part it would be passing data from the DVD, slicing off timecodes and durations, and re-embedding the CC into the video. However, because it is the broadcast-style CC, it doesn't form a good replacement for subtitles in any form.

The main reason I went investigating is that there was a bit of confusion on what subtitle/CC support in QT meant, and not only did I want to cut through it, I wanted to see if it was possible to use subpicture-based subtitles using QT's newfound subtitle support. Unfortunately, it looks like it is merely a very strict timed text track format for subtitles, and broadcast-style support for CC (which is why some blogs are misinformed thinking that turning CC on is the only way to get subtitles on the iPhone or iTunes).

I mostly posted this to point out that if someone wanted to pursue CC support for iTunes 7.6, it would work, and be worth it. However, I don't think there is any good solution that includes Handbrake for subtitles yet. Vobsub export support is about the best idea that makes sense, since the user can OCR the Vobsub output and remux it with another tool.
kneeslasher
Experienced
Posts: 85
Joined: Tue Mar 06, 2007 8:40 pm

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by kneeslasher »

CCs do have at least one advantage over OCR's VobSubs: no OCR mistakes. As pointed out, it should just be a connection of DVD CCs -> SCC CCs -> QuickTime CCs. I've tried doing exactly this in a very long and roundabout way using McPoodle's tools and a QuickTime importer for SCC CCs (see http://forum.handbrake.fr/viewtopic.php?f=6&t=4172) but the method is long, tortuous and subject to non-function for no apparent reason. I have gotten it to work for limited cases, proving that it can technically be done, i.e., produce a (selectable) CCed m4v for the iPod/iPhone from a CCed DVD.
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by Cyander »

Yes, but I also want to make it clear it isn't a solution for those of us trying to rip foreign films. :)

It is a good solution for those that want or need CC in English-speaking countries though.
maubp
Posts: 41
Joined: Mon Jun 18, 2007 10:42 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by maubp »

Does anyone know how common CC is on R1 DVDs (both movies and TV shows)? How about non-R1, where I would guess its going to be very rare?
rhester
Veteran User
Posts: 2888
Joined: Tue Apr 18, 2006 10:24 pm

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by rhester »

In my personal experience, CCs are presented on R1 discs only when same-language DVD subs are not present. YMMV.

Rodney
donaciano
Posts: 3
Joined: Mon Sep 24, 2007 5:10 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by donaciano »

As to how common CC are on DVDs. Well basically every TV series set has complete CC included. Since 2006 they are required on basically all TV shows. In the past many shows were exempt but now it's even more common. This is great for me as a deaf person, I'll give you an example: Years ago on Ebay I snagged a complete Twilight Zone original series collection. Yaaaay happy day, since I've seen them on TV and they had CC so I'm good right? Nope. The DVD people didn't transfer the CC so I was stuck without. Spent a while digging around subtitle and transcript forums, I'd print them out and read along with the show, pausing often. Weee fun. I eventually sold the set, as the manufacturer did not want to send me any sort of transcript and there wasn't anything I could do about it. Recently the set has been re-released and now includes CC! :-D YES!! So I mean I've got TZ, MacGyver, Firefly, The Triangle, Freaks and Geeks and others just being played off the DVD. I'm just trying to say that CC data is on a LOT of DVDs out there and many don't have subtitles. Usually a movie has subtitles even in several languages and TV series will only have CC. If the DVD makers are cheapskates then the DVD has subtitles they call "Captions for the deaf/HH" but it's not a CC, it's just a label they put on it so they're legally compliant. I think this is part of the reason many confuse subtitles and CC.

Would be really nice to be able to rip the CC as easily as the subtitles. Look I'm willing to pay a bounty even if I can get this somehow, and I know others would too. It's just sort of depressing when things like 5.1 sound are focused on when nobody depends on them, and CC is needed by others. But I mean seriously I can't complain since all in all it's just entertainment but I hope eventually iTunes movies and shows will ALL have CC data included. I think that having a simple open-source way to do it will embarrass Apple and the studios into getting their stuff together. After all they proclaim their support for "Universal Access" and CC support in iTunes and QT but there's like 3 actual videos I've found that use it.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by jbrjake »

Guys, there is absolutely no need to have this conversation *again*. It was done to death in the Ponies forum.

If you are not a developer, you should not be posting here. Please don't make me lock this thread.
eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by eddyg »

donaciano wrote: Would be really nice to be able to rip the CC as easily as the subtitles. Look I'm willing to pay a bounty even if I can get this somehow, and I know others would too. It's just sort of depressing when things like 5.1 sound are focused on when nobody depends on them, and CC is needed by others. But I mean seriously I can't complain since all in all it's just entertainment but I hope eventually iTunes movies and shows will ALL have CC data included. I think that having a simple open-source way to do it will embarrass Apple and the studios into getting their stuff together. After all they proclaim their support for "Universal Access" and CC support in iTunes and QT but there's like 3 actual videos I've found that use it.
Try out the latest snap-shot for EIA-608 to MKV and MP4 Soft Subtitles.

Next snapshot will have SRT to MKV and MP4 Soft Subtitles, as well as EIA-608 from DVB-T.

Still up in the air whether to allow EIA-608 pass-through to MP4 as a timed text track, i.e. Apple Closed Captions (clcp track). Probably one day. Then you'll be cable to select CC pass-through as either CC or subtitle. As Cyander says we do have some reference code from Apple:

http://developer.apple.com/samplecode/C ... nImporter/

So it's just a matter of reinterpreting that into a structure that HB can use and using the mp4v2 library, and then passing through the EIA-608 data untouched, grouped by timestamp I guess.

It looks like two new tracks, a clcp one for the EIA-608 data, and a timecode track. I haven't figured out the timecode track as yet however, and don't have time right now, looks like it records the time of the first and last caption.

I also am not sure about how the sample data is written, I was confused by the difference in sampleData containing the samples and the sampleDesc, I guess I'd figure that out if I delved into the APIs.

Cheers, Ed.
User avatar
Ritsuka
HandBrake Team
Posts: 1643
Joined: Fri Jan 12, 2007 11:29 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by Ritsuka »

Here's a patch for mp4v2 I wrote to se in subler.
http://handbrake.fr/pastebin/pastebin.php?show=718

Code: Select all

Index: include/mp4v2/general.h
===================================================================
--- include/mp4v2/general.h	(revision 359)
+++ include/mp4v2/general.h	(working copy)
@@ -75,6 +75,8 @@
 #define MP4_CNTL_TRACK_TYPE     "cntl"  /**< Constant: control track. */
 #define MP4_TEXT_TRACK_TYPE     "text"  /**< Constant: text track. */
 #define MP4_SUBTITLE_TRACK_TYPE "sbtl"  /**< Constant: subtitle track. */
+#define MP4_CC_TRACK_TYPE       "clcp"  /**< Constant: closed captions track. */
+
 /*
  * This second set of track types should be created
  * via MP4AddSystemsTrack(type)
Index: include/mp4v2/track.h
===================================================================
--- include/mp4v2/track.h	(revision 359)
+++ include/mp4v2/track.h	(working copy)
@@ -309,7 +309,27 @@
     uint16_t      width,
     uint16_t      height );
 
+/** Add a closed caption track.
+ *
+ *  MP4AddCCTrack adds a closed caption track track to the mp4 file. MP4WriteSample()
+ *  can then be used to add the desired video samples.
+ *
+ *  @param hFile handle of file for operation.
+ *  @param timeScale the timescale in ticks per second of the track.
+ *  @param width specifies the video frame width in pixels.
+ *  @param height specifies the video frame height in pixels.
+ *
+ *  @return On success, the track-id of the new track.
+ *      On error, #MP4_INVALID_TRACK_ID.
+ */
 MP4V2_EXPORT
+MP4TrackId MP4AddCCTrack(
+    MP4FileHandle hFile,
+    uint32_t      timeScale,
+    uint16_t      width,
+    uint16_t      height );
+
+MP4V2_EXPORT
 MP4TrackId MP4AddPixelAspectRatio(
     MP4FileHandle hFile,
     MP4TrackId    refTrackId,
Index: GNUmakefile.am
===================================================================
--- GNUmakefile.am	(revision 359)
+++ GNUmakefile.am	(working copy)
@@ -16,6 +16,7 @@
     src/atom_amr.cpp                     \
     src/atom_avc1.cpp                    \
     src/atom_avcC.cpp                    \
+    src/atom_c608.cpp                    \
     src/atom_chpl.cpp                    \
     src/atom_colr.cpp                    \
     src/atom_d263.cpp                    \
Index: src/atom_c608.cpp
===================================================================
--- src/atom_c608.cpp	(revision 0)
+++ src/atom_c608.cpp	(revision 0)
@@ -0,0 +1,46 @@
+/*
+ * The contents of this file are subject to the Mozilla Public
+ * License Version 1.1 (the "License"); you may not use this file
+ * except in compliance with the License. You may obtain a copy of
+ * the License at http://www.mozilla.org/MPL/
+ *
+ * Software distributed under the License is distributed on an "AS
+ * IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * rights and limitations under the License.
+ *
+ * The Original Code is MPEG4IP.
+ *
+ * The Initial Developer of the Original Code is Cisco Systems Inc.
+ * Portions created by Cisco Systems Inc. are
+ * Copyright (C) Cisco Systems Inc. 2001.  All Rights Reserved.
+ */
+
+#include "src/impl.h"
+
+namespace mp4v2 {
+namespace impl {
+
+///////////////////////////////////////////////////////////////////////////////
+
+MP4C608Atom::MP4C608Atom()
+        : MP4Atom("c608")
+{
+    AddReserved("reserved1", 4); /* 0 */
+    AddReserved("reserved2", 2); /* 1 */
+
+    AddProperty(new MP4Integer16Property("dataReferenceIndex")); /* 2 */
+}
+
+void MP4C608Atom::Generate()
+{
+    // generate children
+    MP4Atom::Generate();
+
+    ((MP4Integer16Property*)m_pProperties[2])->SetValue(1);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+}
+} // namespace mp4v2::impl
Index: src/mp4file.h
===================================================================
--- src/mp4file.h	(revision 359)
+++ src/mp4file.h	(working copy)
@@ -387,6 +387,10 @@
     MP4TrackId AddSubtitleTrack(uint32_t timescale,
                                 uint16_t width,
                                 uint16_t height);
+    
+    MP4TrackId AddCCTrack(uint32_t timeScale,
+                          uint16_t width,
+                          uint16_t height);
 
     MP4TrackId AddPixelAspectRatio(MP4TrackId trackId, uint32_t hSpacing, uint32_t vSpacing);
     MP4TrackId AddColr(MP4TrackId trackId, uint16_t pri, uint16_t tran, uint16_t mat);
Index: src/mp4atom.cpp
===================================================================
--- src/mp4atom.cpp	(revision 359)
+++ src/mp4atom.cpp	(working copy)
@@ -849,6 +849,8 @@
             break;
 
         case 'c':
+            if( ATOMID(type) == ATOMID("c608") )
+                return new MP4C608Atom();
             if( ATOMID(type) == ATOMID("chap") )
                 return new MP4TrefTypeAtom( type );
             if( ATOMID(type) == ATOMID("chpl") )
Index: src/mp4.cpp
===================================================================
--- src/mp4.cpp	(revision 359)
+++ src/mp4.cpp	(working copy)
@@ -1173,6 +1173,23 @@
         }
         return MP4_INVALID_TRACK_ID;
     }
+    
+    MP4TrackId MP4AddCCTrack(MP4FileHandle hFile,
+                          uint32_t timeScale,
+                          uint16_t width,
+                          uint16_t height)
+    {
+        if (MP4_IS_VALID_FILE_HANDLE(hFile)) {
+            try {
+                return ((MP4File*)hFile)->AddCCTrack(timeScale, width, height);
+            }
+            catch (MP4Error* e) {
+                PRINT_ERROR(e);
+                delete e;
+            }
+        }
+        return MP4_INVALID_TRACK_ID;
+    }    
 
     MP4TrackId MP4AddChapterTextTrack(
         MP4FileHandle hFile, MP4TrackId refTrackId, uint32_t timescale)
Index: src/atoms.h
===================================================================
--- src/atoms.h	(revision 359)
+++ src/atoms.h	(working copy)
@@ -398,6 +398,12 @@
     void Generate();
 };
 
+class MP4C608Atom : public MP4Atom {
+public:
+    MP4C608Atom();
+    void Generate();
+};
+
 class MP4FtabAtom : public MP4Atom {
 public:
     MP4FtabAtom();
Index: src/mp4file.cpp
===================================================================
--- src/mp4file.cpp	(revision 359)
+++ src/mp4file.cpp	(working copy)
@@ -2095,6 +2095,31 @@
     return trackId;
 }
 
+MP4TrackId MP4File::AddCCTrack(uint32_t timeScale,
+                               uint16_t width,
+                               uint16_t height)
+    {
+        MP4TrackId trackId =
+        AddTrack(MP4_CC_TRACK_TYPE, timeScale);
+
+        InsertChildAtom(MakeTrackName(trackId, "mdia.minf"), "nmhd", 0);
+
+        AddChildAtom(MakeTrackName(trackId, "mdia.minf.stbl.stsd"), "c608");
+
+        SetTrackFloatProperty(trackId, "tkhd.width", width);
+        SetTrackFloatProperty(trackId, "tkhd.height", height);
+
+        // stsd is a unique beast in that it has a count of the number
+        // of child atoms that needs to be incremented after we add the tx3g atom
+        MP4Integer32Property* pStsdCountProperty;
+        FindIntegerProperty(
+                            MakeTrackName(trackId, "mdia.minf.stbl.stsd.entryCount"),
+                            (MP4Property**)&pStsdCountProperty);
+        pStsdCountProperty->IncrementValue();
+
+        return trackId;
+    }    
+
 MP4TrackId MP4File::AddChapterTextTrack(MP4TrackId refTrackId, uint32_t timescale)
 {
     // validate reference track id
The samples have to be wrapped inside a cdat atom. I don't think the timecode track is necessary, I created some without it and they worked.
But I didn't really do much testing.
eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: QT Subtitles & Closed Captioning (Mostly Bad News)

Post by eddyg »

Thanks Ritsuka, that will come in very handy. The cdat atom looks pretty straightforward, just need to collect all cc's with the same time stamp and push them out. Because of the way that cc's are implemnted in HB it should be trivial to do pass-through.

Cheers Ed
Post Reply