Adding SSA Subtitle Support

[davidfstr]

I am starting work on adding SSA subtitle support to HandBrake. This is a major new feature, which touches a lot of subsystems. As such, I'd like some feedback on the approach I have in mind for doing this.

My ultimate motivation for making this change is so that I can transcode anime (which primarily has SSA subtitles) to be playable on my iPod Touch. Of course my work should be more generally useful for other purposes.

First I will need to add support for subtitles originating from file-inputs: The hb_stream -> hb_title conversion logic in ffmpeg_title_scan [stream.c] must be extended to convert AVStreams of type SUBTITLE to hb_subtitle objects attached to the single output hb_title object.

I don't think the demuxers (or the reader logic) need updating.

Then there are three ways that SSA subtitles can be pushed to the output:

1. Render on the video track, in a way similar to VOB (DVD bitmap) subtitles

• subtitle->fifo_in ---[hb_decssasub*]---> (banded 8-bit YAUV)
  subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
  job->fifo_sync ---[hb_render]--->
  job->fifo_render ---[hb_encFOOvideo]--->
  job->fifo_mpeg4

2. Downconvert to styled text subtitles (AKA UTF-8 subtitles) - losing information such as text size, position, and color

• subtitle->fifo_in ---[hb_decssasub*]---> (styled text subtitles, with limited formatting: <b>, <i>, <u>)
  subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
  subtitle->fifo_out

3. Passthru to target container - loses no information, but requires the target container to support SSA subtitles

• subtitle->fifo_in ---[hb_decssasub*]---> (unchanged)
  subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
  subtitle->fifo_out

In the case of output method #3, the muxers would need to be updated as well.

• The MKV muxer (hb_mux_mkv) is easy to update to support SSA subtitles, since it supports MK_SUBTITLE_SSA, MK_SUBTITLE_ASS, and even MK_SUBTITLE_USF subtitle tracks natively.
• The MP4 muxer (hb_mux_mp4) is more tricky. I'm not even sure if the container supports SSA subtitles natively. Further investigation is needed here.
• The other muxers, OGM (hb_mux_ogm) and AVI (hb_mux_avi), do not appear to be supported anymore, judging by their omission from the MacGUI.

I can write the rendering logic for output method #1 using VLC's subtitle rendering code as a guide. VLC's license (GPL) is compatible with HandBrake's (also GPL).

Then there exists the matter of updating the CLI and the GUIs.

• Currently the CLI supports the selection of only two subtitle output methods, burned (AKA rendered) or non-burned (AKA passthru), whereas now I've provided three above. I'll probably implement method #1 as the burned method and either #2 or #3 as the passthru method, depending on whether the output container supports SSA subtitles natively.
• The MacGUI also supports the selection of the same subtitle output methods: burned and non-burned. Therefore the output-method-selection logic should be the same.
• I don't have any experience with the other GUIs.

Since I'm new to this project, I'm not familiar with the current set of contributors and their areas of expertise. Is there a list somewhere that has this information?
Update: For that last question I just found this: http://trac.handbrake.fr/browser/trunk/AUTHORS

[Ritsuka]

There is no support for ssa in mp4, so you should just feed it the UTF-8 version that will be converted to tx3g inside the mp4 muxer.

[davidfstr]

Here's a first stab at adding support for subtitles originating from file inputs. This is not commit quality yet, as it still has a lot of debug code floating around. Not to mention that it doesn't work completely (see below).

http://handbrake.fr/pastebin/pastebin.php?show=1346

Changes:

• Extended ffmpeg_title_scan() [stream.c] to convert AVStreams corresponding to either DVD (aka VOB) or TEXT (aka CC, SRT) subtitles to hb_subtitle tracks. (Right now I'm ignoring all other subtitle tracks, including SSA.)
• Added the hb_decutf8sub work-object and modified the transcoding pipeline to recognize it.

I am testing against the following file, which has both an SSA and an SRT (TEXT) subtitle track:
http://www.2shared.com/file/12659711/22 ... _DVD_.html

I am concentrating on getting the TEXT track working first, since HandBrake already handles them in the muxer logic.

One problem I'm running into is that I can't figure out the proper duration for subtitle packets (i.e. hb_buffer instances). The trick is that the FFMPEG file-based demuxer in ffmpeg_read() [stream.c] does not set the 'stop' value for the hb_buffer packets it generates; it only sets 'data', 'size', and 'start'. Unfortunately there isn't an obvious way to determine the correct value for 'stop', as the 'duration' member of the corresponding AVPacket doesn't seem to be set. I think the input MKV container must have this information somewhere, as VLC is able to read it and display the subtitles correctly, but I can't seem to coerce it out of libavcodec.

Anyone with MKV or libavcodec experience able to shed some light on this?

[davidfstr]

Apparently the MKV demuxer in libavcodec saves the duration of UTF-8 (TEXT) subtitle packets in the 'convergence_duration' member of the AVPackets it generates. It looks like the durations of other subtitle packets are lost, but I have not tested that.

So now UTF-8 subtitles can be passed thru without issue, at least from MKV sources. Working on VOB passthru now.

[davidfstr]

Stage one complete: general support for subtitles from file-inputs
http://forum.handbrake.fr/viewtopic.php?f=4&t=16099
http://trac.handbrake.fr/changeset/3283

Next steps are:

• get VOB passthru from file-inputs working
• write SSA->UTF-8 decoder, so that basic SSA passthru works

Then some optimizations:

• TX3G->TX3G when the destination container supports TX3G natively (i.e. only MP4 for now)
• SSA->SSA when the destination container supports SSA natively (i.e. only MKV for now)

Bonus:

• SSA->TX3G that preserves more style information than the SSA->UTF-8 decoder
• burn-in support for SSA using libass

[Starhawk]

This is fantastic. I'd be happy with simply burning in SSA subtitles, which I think is what you are describing in method #1.

[davidfstr]

Stage two complete: HandBrake can now transcode all subtitle formats that it can output (VOB, UTF-8, TX3G).
http://forum.handbrake.fr/viewtopic.php?f=4&t=16267
http://trac.handbrake.fr/changeset/3308

Next step:

• write SSA->UTF-8 decoder, so that basic SSA passthru works

Then I'll get SSA burn-in working with libass. That will be tricky though, as some modifications to the subtitle handling architecture will be necessary:

• Various parts of the code assume that all PICTURESUB subtitles are necessarily VOB subtitles, which will no longer be the case.
• The output format (PICTURESUB vs. TEXTSUB) will differ depending on the config.dest (RENDERSUB vs. PASSTHRUSUB). These two pieces of data were not correlated in the past.
• The encvobsub work-object should be killed once and for all, as it performs no action and complicates the architecture.

[JohnAStebbins]

davidfstr, the original idea for encvobsub was the eventual support of BD PGS-->VOBSUB translation. The idea was that the only supported bitmap subtitles in mkv were vobsubs, so transation would be necessary. I believe mkv now also supports PGS, though their documentation doesn't yet show it. But there may be other bitmap formats that we want to translate to VOBSUB for the purpose of laying down soft bitmap subtitle tracks.

[mkelley]

Please don't get rid of code that may one day allow vobsubs in MKVs -- my output device can only display them that way (so I have hope that Handbrake may one day allow me to convert my blu-ray subs without me having to use outside tools to add them).

[davidfstr]

davidfstr, the original idea for encvobsub was the eventual support of BD PGS-->VOBSUB translation. The idea was that the only supported bitmap subtitles in mkv were vobsubs, so transation would be necessary. I believe mkv now also supports PGS, though their documentation doesn't yet show it. But there may be other bitmap formats that we want to translate to VOBSUB for the purpose of laying down soft bitmap subtitle tracks.

I would expect that any BD PGS -> VOBSUB translation would be done by a dec-pgs-sub work-object, or similar. This wouldn't need the enc-vob-sub work-object.

In fact, the only reason you would need an enc*sub work-object in addition to the existing dec*sub work-objects would be if there was a multi-step subtitle conversion process, with an intermediate subtitle format. Seems unlikely to me.

Please don't get rid of code that may one day allow vobsubs in MKVs

We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.

[JohnAStebbins]

The general workflow in handbrake is decode->sync->render->encode. I was following that paradigm. A bitmap subtitle would be decoded into some intermediate format. If it's to be burned in, that happens in render (note that this is the current workflow for burning vobsubs. but the intermediate format is vobsub). If it is to be encoded to an alternate bitmap format, that happens in encode.

EDIT: I should mention that we do bypass stages in the pipeline when they are not needed. The pipeline gets set up in work.c. I didn't bypass encvobsub because I was lazy.

[mkelley]

Please don't get rid of code that may one day allow vobsubs in MKVs

We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.

Then I'm confused (which is my normal state anyway).

I guess what I need Handbrake to do then is to support the PGS that is in the M2TS file I can get from my blu-ray rips. Right now I can convert them to IDX using BDSup2Sub (after first demuxing the stream using tsmuxer) and then I remux them back into the Handbrake encoded MKV so I have my blu-ray subs, but it would be nice if Handbrake could do this for me (which is to say, read the PGS as a subtitle track and do whatever magic it has to do to make it a soft sub -- I don't need/want hard subs, although I do at times need to select only the forced subs).

I'm guessing whatever you're doing doesn't have anything to do with this, though (although if I understand correctly, and I may not, I could use the IDX file I get from tsmuxer/BDSup2Sub and skip the mkvmerge step, which is okay but not really all that useful to me. The step I really want to skip is demuxing with tsmuxer in the first place)

[kfreeb]

Please don't get rid of code that may one day allow vobsubs in MKVs

We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.

Then I'm confused (which is my normal state anyway).

I guess what I need Handbrake to do then is to support the PGS that is in the M2TS file I can get from my blu-ray rips. Right now I can convert them to IDX using BDSup2Sub (after first demuxing the stream using tsmuxer) and then I remux them back into the Handbrake encoded MKV so I have my blu-ray subs, but it would be nice if Handbrake could do this for me (which is to say, read the PGS as a subtitle track and do whatever magic it has to do to make it a soft sub -- I don't need/want hard subs, although I do at times need to select only the forced subs).

I'm guessing whatever you're doing doesn't have anything to do with this, though (although if I understand correctly, and I may not, I could use the IDX file I get from tsmuxer/BDSup2Sub and skip the mkvmerge step, which is okay but not really all that useful to me. The step I really want to skip is demuxing with tsmuxer in the first place)

I have ripped about 150 blu-rays so far, and they all use PGS subtitles, so having handbrake directly reading and allowing the burn in of them would be the way to go. I personally do the work around above as well, but only for those shows that need forced subtitles burned in.

One blu-ray title that uses nothing but subtitling would be "Apocalypto" and one that uses forced subtitling, (only in limited way) would be "Avatar", hope this may point you in the right direction to a couple of titles you could use.

[kfreeb]

Does anyone know if there has been anymore movement on direct support of PGS subtitles in Handbrake?

[Deleted User 11865]

Does anyone know if there has been anymore movement on direct support of PGS subtitles in Handbrake?

It's planned, but no one is currently working on it.

[kfreeb]

Nice to know.

Thanks!