H.264/AAC-HE Transport Streams

Developer discussion and patch submissions only!
Forum rules
This forum is for developer discussion and patch submission only.
eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

H.264/AAC-HE Transport Streams

Postby eddyg » Wed Apr 09, 2008 6:50 am

Hi,

I was playing around with opening a HD transport stream containing H.264 video with AAC-HE audio. This is what HandBrake thinks about it:

I'm thinking that what I need to do to get this to work is figure out how to identify H.264 video, and just pass it through, how to identify AAC-HE audio, and pass it to FAAD2 for decoding and then encode as per usual using faac.

Does all of this sound doable?

Code: Select all

[18:45:55] file is MPEG Transport Stream
[18:45:55] hb_ts_stream_find_pids - found the following PIDS
[18:45:55]     Video PIDS :
[18:45:55]       0xffffffff (-1)
[18:45:55]       0xfa (250)
[18:45:55]       0x12c (300)
[18:45:55]       0x3ea (1002)
[18:45:55]       0x3fc (1020)
[18:45:55]       0x3fc (1020)
[18:45:55]       0x20 (32)
[18:45:55]       0xfa (250)
[18:45:55]       0x12c (300)
[18:45:55]       0x3ea (1002)
[18:45:55]       0x3fc (1020)
[18:45:55]       0x20 (32)
[18:45:55]       0xfa (250)
[18:45:55]       0x12c (300)
[18:45:55]       0x3ea (1002)
[18:45:55]       0x3fc (1020)
[18:45:55]       0x420 (1056)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x0 (0)
[18:45:55]       0x1b00 (6912)
[18:45:55]       0xb11 (2833)
[18:45:55]       0x606 (1542)
[18:45:55]       0x1b05 (6917)
[18:45:55]       0xb11 (2833)
[18:45:55]       0x506 (1286)
[18:45:55]       0x111b (4379)
[18:45:55]     Audio PIDS :
[18:45:55]       0xfa (250)
[18:45:55]       0x12c (300)
[18:45:55]       0x3ea (1002)
[18:45:55]       0x3fc (1020)
[18:45:55] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:55] hb_sample_pts: couldn't find video packet near 534542
[18:45:55] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:55] hb_sample_pts: couldn't find video packet near 1603627
[18:45:55] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:55] hb_sample_pts: couldn't find video packet near 2672712
[18:45:55] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:55] hb_sample_pts: couldn't find video packet near 3741797
[18:45:55] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:55] hb_sample_pts: couldn't find video packet near 4810882
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 5879967
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 6949052
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 8018137
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 9087222
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 10156307
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 11225392
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 12294477
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 13363562
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 14432647
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 15501732
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0xffffffff
[18:45:56] hb_sample_pts: couldn't find video packet near 16570817
[18:45:56] transport stream pid 0xfa (type 0x1b) isn't audio
[18:45:56] transport stream pid 0x12c (type 0x11) is MPEG audio id 0xc1
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0x3ea
[18:45:56] transport stream pid 0x3ea (type 0xb) isn't audio
[18:45:56] hb_ts_stream_getPEStype: EOF while searching for PID 0x3fc
[18:45:56] transport stream pid 0x3fc (type 0x6) isn't audio
Last edited by eddyg on Fri Apr 11, 2008 12:54 am, edited 1 time in total.

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Thu Apr 10, 2008 10:28 pm

FYI. Here are the stream types for the programs in the program map table that I'm seeing when MPEG-4 Video and Audio are encapsulated within an MPEG Transport Stream:

Stream type 0x1b, 27
Stream type 0x11, 17
Stream type 0xb, 11
Stream type 0x6, 6

Now these are supposed to match ISO/IEC 13818-1:2000 Table 2-29, but it doesn't make much sense to me.

0x11 is ISO/IEC 14496-3 Audio with the LATM transport syntax as defined in ISO/IEC 14496-3 / AMD 1 (MPEG-4 audio) which is OK.

But I don't know which of the rest is supposed to be carrying the MPEG-4/H.264 Video.

VLC is decoding this fine - so maybe I should look at that.

What I want to do initially is modify stream.c to do a proper output of the streams and what they are as part of the scan. Once we can identify the MPEG-4 reliably we can move on to extracting it and sending it into HB for processing.

Cheers, Ed.

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Thu Apr 10, 2008 11:45 pm

So much for the standard - I looked at VLC and 0x1b is H.264:

So this is making sense now, the other streams (in various test streams that I have) are subtitles and program guide which we can discard for now.

Code within VLC ts.c:

Code: Select all

    switch( i_stream_type )
    {
        case 0x01:  /* MPEG-1 video */
        case 0x02:  /* MPEG-2 video */
        case 0x80:  /* MPEG-2 MOTO video */
            es_format_Init( fmt, VIDEO_ES, VLC_FOURCC( 'm', 'p', 'g', 'v' ) );
            break;
        case 0x03:  /* MPEG-1 audio */
        case 0x04:  /* MPEG-2 audio */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'm', 'p', 'g', 'a' ) );
            break;
        case 0x11:  /* MPEG4 (audio) */
        case 0x0f:  /* ISO/IEC 13818-7 Audio with ADTS transport syntax */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'm', 'p', '4', 'a' ) );
            break;
        case 0x10:  /* MPEG4 (video) */
            es_format_Init( fmt, VIDEO_ES, VLC_FOURCC( 'm', 'p', '4', 'v' ) );
            pid->es->b_gather = VLC_TRUE;
            break;
        case 0x1B:  /* H264 <- check transport syntax/needed descriptor */
            es_format_Init( fmt, VIDEO_ES, VLC_FOURCC( 'h', '2', '6', '4' ) );
            break;

        case 0x81:  /* A52 (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'a', '5', '2', ' ' ) );
            break;
        case 0x82:  /* DVD_SPU (sub) */
            es_format_Init( fmt, SPU_ES, VLC_FOURCC( 's', 'p', 'u', ' ' ) );
            break;
        case 0x83:  /* LPCM (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'l', 'p', 'c', 'm' ) );
            break;
        case 0x84:  /* SDDS (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 's', 'd', 'd', 's' ) );
            break;
        case 0x85:  /* DTS (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'd', 't', 's', ' ' ) );
            break;

        case 0x91:  /* A52 vls (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 'a', '5', '2', 'b' ) );
            break;
        case 0x92:  /* DVD_SPU vls (sub) */
            es_format_Init( fmt, SPU_ES, VLC_FOURCC( 's', 'p', 'u', 'b' ) );
            break;

        case 0x94:  /* SDDS (audio) */
            es_format_Init( fmt, AUDIO_ES, VLC_FOURCC( 's', 'd', 'd', 'b' ) );
            break;

        case 0xa0:  /* MSCODEC vlc (video) (fixed later) */
            es_format_Init( fmt, UNKNOWN_ES, 0 );
            pid->es->b_gather = VLC_TRUE;
            break;

        case 0x06:  /* PES_PRIVATE  (fixed later) */
        case 0x12:  /* MPEG-4 generic (sub/scene/...) (fixed later) */
        case 0xEA:  /* Privately managed ES (VC-1) (fixed later */

van
Veteran User
Posts: 417
Joined: Wed Aug 29, 2007 6:35 am

Re: H.264/AAC-HE Transport Streams

Postby van » Fri Apr 11, 2008 12:42 am

ISO 13818-1 just defines the container & code points that were known at the time it was last updated. Other code points are spread randomly through other ISO standards & various other registries. The AVC HDTV stuff is mostly in ISO 14496-10. Two good references for sorting out transport streams are the DVB BlueBook http://www.dvb.org/technology/standards/a001r7.tm1214r30.dTS101154.v1.8.1.pdf and the ATSC code point registry http://www.atsc.org/standards/Code_Point_Registry.pdf. Be aware that most of the standards are advisory and broadcasters don't follow them (which is why I made the stream.c code basically ignore the type & descriptor codes and figure things out from the elementary stream headers whose format is mandatory).

HandBrake is probably the wrong tool for investigating the bitstreams - it's easy to add a bunch of debugging info to the log that becomes useless noise after you gain understanding. You'll have better luck with dvbsnoop http://dvbsnoop.sourceforge.net/ and/or atscut (don't have a current URL). Both of these are linux tools but can be made to compile on Mac OS X with minor mods. I can send you a tar of an osx modded dvbsnoop source if you want.

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Fri Apr 11, 2008 12:49 am

Thanks Van - I'll take a look at those tools.

I've been meaning to have a chat with you about this stuff.

1) I made a change to stream.c to print out the stream type that Iike to commit and wanted your OK
http://handbrake.pastebin.ca/980427
http://handbrake.pastebin.ca/980434

2) What work are you doing in this area? I wanted to get an ffmpeg decoder onto the H.264 stream and use FAAD2 for the AAC-HE audio into HB. This is because the HD free to air DTT in NZ is all H.264/AAC-HE in MPEG TS. jbrjake mentioned that you were already looking at the H.264 decoder?

Cheers, Ed.

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Fri Apr 11, 2008 12:51 am

van wrote: Be aware that most of the standards are advisory and broadcasters don't follow them (which is why I made the stream.c code basically ignore the type & descriptor codes and figure things out from the elementary stream headers whose format is mandatory).


You probably want me to remove the changes that I made to recognise Video streams by type and instead lump MPEG-4 and H.264 video as Audio instead, and then use the PES for figuring out the type?

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Fri Apr 11, 2008 12:55 am

FYI with the change the new output looks like:

Code: Select all

[12:12:41] hb_ts_stream_find_pids - found the following PIDS
[12:12:41]     Video PIDS :
[12:12:41]       0xfa (250) [Type H.264 Video (0x1b)]
[12:12:41]     Audio PIDS :
[12:12:41]       0x20 (32) [Type Other (0x0)]
[12:12:41]       0x12c (300) [Type Other (0x5)]
[12:12:41]       0x3ea (1002) [Type MPEG-4 Audio (0x11)]
[12:12:41]       0x3fc (1020) [Type Other (0xb)]
[12:12:41] hb_sample_pts: pts 1021731196 at 535612
[12:12:41] hb_sample_pts: pts 1021806796 at 1607212
[12:12:41] hb_sample_pts: pts 1021889596 at 2749500
[12:12:41] hb_sample_pts: pts 1021956196 at 3759436
[12:12:41] hb_sample_pts: pts 1022064196 at 4824456
[12:12:41] hb_sample_pts: pts 1022136196 at 5882520
[12:12:41] hb_sample_pts: pts 1022240596 at 6952428
[12:12:41] hb_sample_pts: pts 1022317996 at 8043204
[12:12:41] hb_sample_pts: pts 1022409796 at 9106908
[12:12:41] hb_sample_pts: pts 1022490796 at 10161964
[12:12:41] hb_sample_pts: pts 1022595196 at 11268908
[12:12:41] hb_sample_pts: pts 1022681596 at 12314940
[12:12:41] hb_sample_pts: pts 1022757196 at 13363792
[12:12:41] hb_sample_pts: pts 1022839996 at 14491792
[12:12:41] hb_sample_pts: pts 1022908396 at 15518084
[12:12:41] hb_sample_pts: pts 1023005596 at 16574456
[12:12:41] hb_ts_stream_getPEStype: EOF while searching for PID 0x20
[12:12:41] transport stream pid 0x20 (type 0x5) isn't audio
[12:12:41] transport stream pid 0x12c (type 0x11) is MPEG audio id 0xc1
[12:12:41] hb_ts_stream_getPEStype: EOF while searching for PID 0x3ea
[12:12:41] transport stream pid 0x3ea (type 0xb) isn't audio
[12:12:42] hb_ts_stream_getPEStype: EOF while searching for PID 0x3fc
[12:12:42] transport stream pid 0x3fc (type 0x6) isn't audio

van
Veteran User
Posts: 417
Joined: Wed Aug 29, 2007 6:35 am

Re: H.264/AAC-HE Transport Streams

Postby van » Fri Apr 11, 2008 7:22 am

eddyg wrote:You probably want me to remove the changes that I made to recognise Video streams by type and instead lump MPEG-4 and H.264 video as Audio instead, and then use the PES for figuring out the type?

I'm not sure what awk intended with the video & audio stream arrays but the standard is pretty clear that there can be at most one "main" video stream together with various aux streams that can be audio, subtitles, teletext, data, etc. There are many places in the code that "know" stream 0 is video. I've been slowly removing the fiction that there's an array of video streams & another of audio streams and moving to a model where there's just an array of streams & entry 0 has to be the main video. Your patch looks ok to me & you don't have to bother about this but don't be surprised when the structure changes.

So I'd stick the h.264 video into video slot 0 like you're doing but put everything else into audio (like your latest patch is doing) so it will get determined from the PES during the scan - experience so far has been that the video type is reliable but the audio is not and we have to look at the PES anyway during the scan to get bitrate, channels & other attributes that aren't in the PMT.

But this will all be academic for a while. Dealing with the AAC audio is trivial since HB has all the infrastructure to support different audio decoders determined at run time but dealing with the h.264 video is a major restructuring of HB's rendering pipeline. Right now that pipeline & all its support infrastructure is hardwired for mpeg2 video & has to be generalized to decode different video types. I've started to work on this but haven't had enough free time to take it very far.

For right now your best bet for turning these bitstreams into something useful is (a very recent version of) ffmpeg. All the 1080i AVC streams I've seen are High profile and require PAFF and Spatial Direct mode in the h.264 decoder. Support for these only got added to ffmpeg a few months ago and didn't start to get reliable until a few weeks ago.

Anyway, I'd vote for you committing your patch.

eddyg
Veteran User
Posts: 798
Joined: Mon Apr 23, 2007 3:34 am

Re: H.264/AAC-HE Transport Streams

Postby eddyg » Fri Apr 11, 2008 7:45 am

Thanks Van.


Return to “Development”