Adding SSA Subtitle Support

Developer discussion and patch submissions only!
Forum rules
This forum is for developer discussion and patch submission only.
Post Reply
davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Adding SSA Subtitle Support

Post by davidfstr » Wed Apr 14, 2010 6:26 am

I am starting work on adding SSA subtitle support to HandBrake. This is a major new feature, which touches a lot of subsystems. As such, I'd like some feedback on the approach I have in mind for doing this.

My ultimate motivation for making this change is so that I can transcode anime (which primarily has SSA subtitles) to be playable on my iPod Touch. Of course my work should be more generally useful for other purposes.

First I will need to add support for subtitles originating from file-inputs: The hb_stream -> hb_title conversion logic in ffmpeg_title_scan [stream.c] must be extended to convert AVStreams of type SUBTITLE to hb_subtitle objects attached to the single output hb_title object.

I don't think the demuxers (or the reader logic) need updating.

Then there are three ways that SSA subtitles can be pushed to the output:

1. Render on the video track, in a way similar to VOB (DVD bitmap) subtitles
  • subtitle->fifo_in ---[hb_decssasub*]---> (banded 8-bit YAUV)
    subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
    job->fifo_sync ---[hb_render]--->
    job->fifo_render ---[hb_encFOOvideo]--->
    job->fifo_mpeg4
2. Downconvert to styled text subtitles (AKA UTF-8 subtitles) - losing information such as text size, position, and color
  • subtitle->fifo_in ---[hb_decssasub*]---> (styled text subtitles, with limited formatting: <b>, <i>, <u>)
    subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
    subtitle->fifo_out
3. Passthru to target container - loses no information, but requires the target container to support SSA subtitles
  • subtitle->fifo_in ---[hb_decssasub*]---> (unchanged)
    subtitle->fifo_raw ---[hb_sync_video*]---> (unchanged)
    subtitle->fifo_out
In the case of output method #3, the muxers would need to be updated as well.
  • The MKV muxer (hb_mux_mkv) is easy to update to support SSA subtitles, since it supports MK_SUBTITLE_SSA, MK_SUBTITLE_ASS, and even MK_SUBTITLE_USF subtitle tracks natively.
  • The MP4 muxer (hb_mux_mp4) is more tricky. I'm not even sure if the container supports SSA subtitles natively. Further investigation is needed here.
  • The other muxers, OGM (hb_mux_ogm) and AVI (hb_mux_avi), do not appear to be supported anymore, judging by their omission from the MacGUI.
I can write the rendering logic for output method #1 using VLC's subtitle rendering code as a guide. VLC's license (GPL) is compatible with HandBrake's (also GPL).

Then there exists the matter of updating the CLI and the GUIs.
  • Currently the CLI supports the selection of only two subtitle output methods, burned (AKA rendered) or non-burned (AKA passthru), whereas now I've provided three above. I'll probably implement method #1 as the burned method and either #2 or #3 as the passthru method, depending on whether the output container supports SSA subtitles natively.
  • The MacGUI also supports the selection of the same subtitle output methods: burned and non-burned. Therefore the output-method-selection logic should be the same.
  • I don't have any experience with the other GUIs.
Since I'm new to this project, I'm not familiar with the current set of contributors and their areas of expertise. Is there a list somewhere that has this information?
Update: For that last question I just found this: http://trac.handbrake.fr/browser/trunk/AUTHORS

User avatar
Ritsuka
HandBrake Team
Posts: 875
Joined: Fri Jan 12, 2007 11:29 am

Re: Adding SSA Subtitle Support

Post by Ritsuka » Wed Apr 14, 2010 9:21 am

There is no support for ssa in mp4, so you should just feed it the UTF-8 version that will be converted to tx3g inside the mp4 muxer.

davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Adding SSA Subtitle Support

Post by davidfstr » Sun Apr 18, 2010 11:41 pm

Here's a first stab at adding support for subtitles originating from file inputs. This is not commit quality yet, as it still has a lot of debug code floating around. Not to mention that it doesn't work completely (see below).

http://handbrake.fr/pastebin/pastebin.php?show=1346

Changes:
  • Extended ffmpeg_title_scan() [stream.c] to convert AVStreams corresponding to either DVD (aka VOB) or TEXT (aka CC, SRT) subtitles to hb_subtitle tracks. (Right now I'm ignoring all other subtitle tracks, including SSA.)
  • Added the hb_decutf8sub work-object and modified the transcoding pipeline to recognize it.
I am testing against the following file, which has both an SSA and an SRT (TEXT) subtitle track:
http://www.2shared.com/file/12659711/22 ... _DVD_.html

I am concentrating on getting the TEXT track working first, since HandBrake already handles them in the muxer logic.

One problem I'm running into is that I can't figure out the proper duration for subtitle packets (i.e. hb_buffer instances). The trick is that the FFMPEG file-based demuxer in ffmpeg_read() [stream.c] does not set the 'stop' value for the hb_buffer packets it generates; it only sets 'data', 'size', and 'start'. Unfortunately there isn't an obvious way to determine the correct value for 'stop', as the 'duration' member of the corresponding AVPacket doesn't seem to be set. I think the input MKV container must have this information somewhere, as VLC is able to read it and display the subtitles correctly, but I can't seem to coerce it out of libavcodec.

Anyone with MKV or libavcodec experience able to shed some light on this?

davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Adding SSA Subtitle Support

Post by davidfstr » Sun Apr 25, 2010 8:36 am

Apparently the MKV demuxer in libavcodec saves the duration of UTF-8 (TEXT) subtitle packets in the 'convergence_duration' member of the AVPackets it generates. It looks like the durations of other subtitle packets are lost, but I have not tested that.

So now UTF-8 subtitles can be passed thru without issue, at least from MKV sources. Working on VOB passthru now.

davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Adding SSA Subtitle Support

Post by davidfstr » Thu May 06, 2010 3:54 am

Stage one complete: general support for subtitles from file-inputs
http://forum.handbrake.fr/viewtopic.php?f=4&t=16099
http://trac.handbrake.fr/changeset/3283

Next steps are:
  • get VOB passthru from file-inputs working
  • write SSA->UTF-8 decoder, so that basic SSA passthru works
Then some optimizations:
  • TX3G->TX3G when the destination container supports TX3G natively (i.e. only MP4 for now)
  • SSA->SSA when the destination container supports SSA natively (i.e. only MKV for now)
Bonus:
  • SSA->TX3G that preserves more style information than the SSA->UTF-8 decoder
  • burn-in support for SSA using libass

Starhawk
Regular User
Posts: 90
Joined: Sun Feb 24, 2008 8:27 pm

Re: Adding SSA Subtitle Support

Post by Starhawk » Thu May 06, 2010 2:06 pm

This is fantastic. I'd be happy with simply burning in SSA subtitles, which I think is what you are describing in method #1.

davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Adding SSA Subtitle Support

Post by davidfstr » Sat May 22, 2010 6:52 am

Stage two complete: HandBrake can now transcode all subtitle formats that it can output (VOB, UTF-8, TX3G).
http://forum.handbrake.fr/viewtopic.php?f=4&t=16267
http://trac.handbrake.fr/changeset/3308

Next step:
  • write SSA->UTF-8 decoder, so that basic SSA passthru works
Then I'll get SSA burn-in working with libass. That will be tricky though, as some modifications to the subtitle handling architecture will be necessary:
  • Various parts of the code assume that all PICTURESUB subtitles are necessarily VOB subtitles, which will no longer be the case.
  • The output format (PICTURESUB vs. TEXTSUB) will differ depending on the config.dest (RENDERSUB vs. PASSTHRUSUB). These two pieces of data were not correlated in the past.
  • The encvobsub work-object should be killed once and for all, as it performs no action and complicates the architecture.

User avatar
JohnAStebbins
HandBrake Team
Posts: 4953
Joined: Sat Feb 09, 2008 7:21 pm

Re: Adding SSA Subtitle Support

Post by JohnAStebbins » Sat May 22, 2010 3:30 pm

davidfstr, the original idea for encvobsub was the eventual support of BD PGS-->VOBSUB translation. The idea was that the only supported bitmap subtitles in mkv were vobsubs, so transation would be necessary. I believe mkv now also supports PGS, though their documentation doesn't yet show it. But there may be other bitmap formats that we want to translate to VOBSUB for the purpose of laying down soft bitmap subtitle tracks.

mkelley
Bright Spark User
Posts: 389
Joined: Fri Dec 25, 2009 2:00 am

Re: Adding SSA Subtitle Support

Post by mkelley » Sat May 22, 2010 8:08 pm

Please don't get rid of code that may one day allow vobsubs in MKVs -- my output device can only display them that way (so I have hope that Handbrake may one day allow me to convert my blu-ray subs without me having to use outside tools to add them).

davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Adding SSA Subtitle Support

Post by davidfstr » Sun May 23, 2010 3:29 am

JohnAStebbins wrote:davidfstr, the original idea for encvobsub was the eventual support of BD PGS-->VOBSUB translation. The idea was that the only supported bitmap subtitles in mkv were vobsubs, so transation would be necessary. I believe mkv now also supports PGS, though their documentation doesn't yet show it. But there may be other bitmap formats that we want to translate to VOBSUB for the purpose of laying down soft bitmap subtitle tracks.
I would expect that any BD PGS -> VOBSUB translation would be done by a dec-pgs-sub work-object, or similar. This wouldn't need the enc-vob-sub work-object.

In fact, the only reason you would need an enc*sub work-object in addition to the existing dec*sub work-objects would be if there was a multi-step subtitle conversion process, with an intermediate subtitle format. Seems unlikely to me.
mkelley wrote:Please don't get rid of code that may one day allow vobsubs in MKVs
We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.

User avatar
JohnAStebbins
HandBrake Team
Posts: 4953
Joined: Sat Feb 09, 2008 7:21 pm

Re: Adding SSA Subtitle Support

Post by JohnAStebbins » Sun May 23, 2010 3:53 am

The general workflow in handbrake is decode->sync->render->encode. I was following that paradigm. A bitmap subtitle would be decoded into some intermediate format. If it's to be burned in, that happens in render (note that this is the current workflow for burning vobsubs. but the intermediate format is vobsub). If it is to be encoded to an alternate bitmap format, that happens in encode.

EDIT: I should mention that we do bypass stages in the pipeline when they are not needed. The pipeline gets set up in work.c. I didn't bypass encvobsub because I was lazy.

mkelley
Bright Spark User
Posts: 389
Joined: Fri Dec 25, 2009 2:00 am

Re: Adding SSA Subtitle Support

Post by mkelley » Sun May 23, 2010 5:44 pm

davidfstr wrote:
mkelley wrote:Please don't get rid of code that may one day allow vobsubs in MKVs
We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.
Then I'm confused (which is my normal state anyway).

I guess what I need Handbrake to do then is to support the PGS that is in the M2TS file I can get from my blu-ray rips. Right now I can convert them to IDX using BDSup2Sub (after first demuxing the stream using tsmuxer) and then I remux them back into the Handbrake encoded MKV so I have my blu-ray subs, but it would be nice if Handbrake could do this for me (which is to say, read the PGS as a subtitle track and do whatever magic it has to do to make it a soft sub -- I don't need/want hard subs, although I do at times need to select only the forced subs).

I'm guessing whatever you're doing doesn't have anything to do with this, though (although if I understand correctly, and I may not, I could use the IDX file I get from tsmuxer/BDSup2Sub and skip the mkvmerge step, which is okay but not really all that useful to me. The step I really want to skip is demuxing with tsmuxer in the first place)

kfreeb
Posts: 62
Joined: Wed Sep 01, 2010 4:10 pm

Re: Adding SSA Subtitle Support

Post by kfreeb » Thu Sep 30, 2010 12:58 am

mkelley wrote:
davidfstr wrote:
mkelley wrote:Please don't get rid of code that may one day allow vobsubs in MKVs
We already support VOB subtitles. :wink:
Don't worry, I just added more support for VOB subtitles - they aren't going away.
Then I'm confused (which is my normal state anyway).

I guess what I need Handbrake to do then is to support the PGS that is in the M2TS file I can get from my blu-ray rips. Right now I can convert them to IDX using BDSup2Sub (after first demuxing the stream using tsmuxer) and then I remux them back into the Handbrake encoded MKV so I have my blu-ray subs, but it would be nice if Handbrake could do this for me (which is to say, read the PGS as a subtitle track and do whatever magic it has to do to make it a soft sub -- I don't need/want hard subs, although I do at times need to select only the forced subs).

I'm guessing whatever you're doing doesn't have anything to do with this, though (although if I understand correctly, and I may not, I could use the IDX file I get from tsmuxer/BDSup2Sub and skip the mkvmerge step, which is okay but not really all that useful to me. The step I really want to skip is demuxing with tsmuxer in the first place)
I have ripped about 150 blu-rays so far, and they all use PGS subtitles, so having handbrake directly reading and allowing the burn in of them would be the way to go. I personally do the work around above as well, but only for those shows that need forced subtitles burned in.

One blu-ray title that uses nothing but subtitling would be "Apocalypto" and one that uses forced subtitling, (only in limited way) would be "Avatar", hope this may point you in the right direction to a couple of titles you could use.

kfreeb
Posts: 62
Joined: Wed Sep 01, 2010 4:10 pm

Re: Adding SSA Subtitle Support

Post by kfreeb » Mon Apr 25, 2011 7:45 pm

Does anyone know if there has been anymore movement on direct support of PGS subtitles in Handbrake?

User avatar
Rodeo
HandBrake Team
Posts: 11517
Joined: Tue Mar 03, 2009 8:55 pm

Re: Adding SSA Subtitle Support

Post by Rodeo » Mon Apr 25, 2011 7:50 pm

kfreeb wrote:Does anyone know if there has been anymore movement on direct support of PGS subtitles in Handbrake?
It's planned, but no one is currently working on it.

kfreeb
Posts: 62
Joined: Wed Sep 01, 2010 4:10 pm

Re: Adding SSA Subtitle Support

Post by kfreeb » Mon Apr 25, 2011 7:57 pm

Nice to know.

Thanks!

Post Reply