MPEG-2 Transport Stream Support

alynpier · Post by **alynpier** » Thu Apr 26, 2007 5:14 am

Thanks for the job, awk ! I was really looking for that; I got EyeTV files too.
When will Handbrake add your stuff ? Because I am not able to test anything, since it is not in HB itself...

awk · Post by **awk** » Thu Apr 26, 2007 10:20 am

alynpier wrote:Thanks for the job, awk ! I was really looking for that; I got EyeTV files too.
When will Handbrake add your stuff ? Because I am not able to test anything, since it is not in HB itself...

I don't have a schedule for when my work will get added to the main code I'm afraid. At this point it's still very fresh and untested by many people beyond me

I anticipate another update to the code today and then some more testing. I'll have to let the lead developers (jbrjake and rhester et al) speak as to when they think it will make a release and when that release might be.

alynpier · Post by **alynpier** » Thu Apr 26, 2007 12:21 pm

OK, thanks, I'll wait...

awk · Post by **awk** » Thu Apr 26, 2007 1:14 pm

At http://awkward.org/handbrake/mpeg2-ts-patch.tar.gz

You'll need to manually add the libhb/stream.c file to the project/makefiles (depending on platform) and then to the libhb and libhb dlib target.

It seems to work OK for me with the transport streams in common use here in the US for digital broadcasting so long as there is only one video stream (ie PMT/PAT filtering has already taken place). MPEG-2 Audio (and LPCM audio) in a transport stream is not supported yet (US broadcasts don't do this so I've not tested it yet. I do have a couple of test clips now though).

The program stream stuff should still work fine (VOB/.mpg files etc.) but there do still seem to be problems with Handbrake's handling of non-AC3 audio. I'm going to take a look at these problems next since I do have some Program Streams with non-AC3 audio (thanks folks).

Some more details are on my blog : http://awkward.org/2007/04/26/transport-stream-support/

jbrjake · Post by **jbrjake** » Thu Apr 26, 2007 1:29 pm

Well I'd like to get this stuff checked in as soon as possible, because this so unbelievably cool.

I'm still having problems getting it to build, though. Same stuff I was getting before--parse errors trying to compile stream.c after I add it to the xcode project. I'm sure it's just idiocy on my part

awk · Post by **awk** » Thu Apr 26, 2007 1:51 pm

jbrjake wrote: I'm still having problems getting it to build, though. Same stuff I was getting before--parse errors trying to compile stream.c after I add it to the xcode project. I'm sure it's just idiocy on my part

Hmmm.... I did a 'resh sync' on a second machine this morning and applied the patches and built fine so I'm not sure what could be causing this. Can you give me the first 5-10 lines of errors and also which version of XCode and OS X etc. you have (just in case there's some funky tools issue here).

jbrjake · Post by **jbrjake** » Thu Apr 26, 2007 1:55 pm

It was just idiocy--I added it to too many targets. Compiled fine, gonna test it now

alynpier · Post by **alynpier** » Thu Apr 26, 2007 1:55 pm

awk wrote:You'll need to manually add the libhb/stream.c file to the project/makefiles (depending on platform) and then to the libhb and libhb dlib target.

I'm far to be able to make that !!!

jbrjake · Post by **jbrjake** » Thu Apr 26, 2007 2:34 pm

1080->720, encoded with HandBrake:

awk · Post by **awk** » Thu Apr 26, 2007 2:51 pm

Nice !

One of the potential uses for this (that only just occured to me) is for people who wish to transfer footage shot with a HDV Camcorder to H.264. Once I fix the MPEG-2 Audio issues you should be able to take a camera stream and transcode it (though getting the stream off the camera is a seperate task

This is actually the most reasonable way for someone to get a true 720p24 H.264 stream that they can test on an Apple TV ! JVC makes a couple of cameras which shoot at 720p24 - I'll try and grab one from work and make some test footage and convert it !

cvk_b · Post by **cvk_b** » Thu Apr 26, 2007 8:34 pm

Tried the patch for the first time today. Wow. Thanks. This is big.

tlunde · Post by **tlunde** » Thu Apr 26, 2007 10:25 pm

Mmmmm! Can't wait to use this new feature with a HD HomeRun. Since mythtv's frontend isn't ready (yet?) for AppleTV, this would be the next best thing.

maurj · Post by **maurj** » Fri Apr 27, 2007 10:32 am

Hi awk,

I've just tried the patch against rev 553, and it applied cleanly. Whilst I can't "make", I can build the HandBrake target in Xcode with no problems.

However, when I launch HandBrake and select an MPEG transport stream via the "DVD Folder / image" browse button, I get an error of "No valid title found". This is a PAL transport stream recorded and exported by EyeTV 2.4. The file will open in MPEG StreamClip fine, and has one Video PID and one audio PID of MP2 stereo. Aha - maybe that's why? Because it only has MP2 audio?

BTW, the current version of HandBrake has something up with its DVD MPEG2 extraction, resulting in stuttery audio. If you can work out what the problem is as part of this work, then that's an extra bonus and saves me investigating it

- maurj.

awk · Post by **awk** » Fri Apr 27, 2007 10:39 am

maurj wrote:Hi awk,

I've just tried the patch against rev 553, and it applied cleanly. Whilst I can't "make", I can build the HandBrake target in Xcode with no problems.

I've only been using XCode myself, so this isn't too much of a surprise. I though make just launched xcodebuild (which uses the projects), I'll look into it for the next update.

maurj wrote:However, when I launch HandBrake and select an MPEG transport stream via the "DVD Folder / image" browse button, I get an error of "No valid title found". This is a PAL transport stream recorded and exported by EyeTV 2.4. The file will open in MPEG StreamClip fine, and has one Video PID and one audio PID of MP2 stereo. Aha - maybe that's why? Because it only has MP2 audio?

Yep - if it's a transport stream with MP2 stereo audio that's not going to report a valid title - a program stream should report a valid title (I think), but I've not tested it. I have a couple of transport stream clips with MP2 audio but they're hi-def from Sony & JVC cameras, if you can make available a short example clip for me I'll add it to my test matrix (only needs to be about 10megabytes or so - PM me the link if you wish).

maurj wrote:BTW, the current version of HandBrake has something up with its DVD MPEG2 extraction, resulting in stuttery audio. If you can work out what the problem is as part of this work, then that's an extra bonus and saves me investigating it

- maurj.

There is a change I made - in sync.c I think - for handling MPEG2 stream packets with no PTS information. If you review the patch you should see it. That change could be tested/committed separately if it seems to fix the problem - basically if there's no PTS for a packet then I calculate the expected one and use that.

maurj · Post by **maurj** » Fri Apr 27, 2007 11:04 am

Hi awk,

I've only been using XCode myself, so this isn't too much of a surprise. I though make just launched xcodebuild (which uses the projects), I'll look into it for the next update.

Me too (but I've never really looked into it).

Yep - if it's a transport stream with MP2 stereo audio that's not going to report a valid title - a program stream should report a valid title (I think), but I've not tested it.

My error - it's a program stream, not a transport stream. I've PM'd you with a download link for a sample clip.

There is a change I made - in sync.c I think - for handling MPEG2 stream packets with no PTS information. If you review the patch you should see it. That change could be tested/committed separately if it seems to fix the problem - basically if there's no PTS for a packet then I calculate the expected one and use that.

Strange thing is, MPEG2 audio used to work. I've not yet done the regression to find out when it broke, but if I had to guess it would be "a long time back". If I get the time, I'll take a look at the change you made, and see if it can help out elsewhere.

Thanks for the work to get this into HandBrake - it's going to be a killer feature!

- maurj.

awk · Post by **awk** » Fri Apr 27, 2007 12:28 pm

maurj wrote:
My error - it's a program stream, not a transport stream. I've PM'd you with a download link for a sample clip.

Ok - that makes sense - I have another test clip (from France) that's also EyeTV and is a program stream. It looks like EyeTV handles any PAT/PMT Filtering itself and just outputs a program stream - no problems.

awk · Post by **awk** » Wed May 09, 2007 3:04 am

kogyaru wrote: The patch correctly reads in VOB files, either named .vob or .mpg, but it doesn't seem to work with SVCD .mpg files. I'm interested in that as I've got a considerable archive of SVCDs.

I took a little look at this tonight. Unfortunately it doesn't look good

The problem is that SVCD MPEG Elementary Stream Packets appear to have a length of 2300 (or so - the exact value escapes me right now) bytes. This is different from the DVD (and general digital broadcasting) standard of 2028 bytes (plus header data for a 2048 byte total length).

Handbrake's Program Stream Demuxer is pretty heavily skewed towards a compile time fixed packet length - it can be changed, but not dynamically - and there's little to no way to reflect 'left over data' back to the reading processes. So there's not even really an option right now for just using a larger buffer size and hoping that will do for DVDs.

Updating the program stream demuxing code to be more flexible would be a great thing but it's outside of my time right now I'm afraid. If others want to take it up I hope I can be of assistance and give some guidance but that's about all I can afford.

Sorry I don't have more positive news

kogyaru · Post by **kogyaru** » Wed May 09, 2007 6:11 pm

awk wrote:Updating the program stream demuxing code to be more flexible would be a great thing but it's outside of my time right now I'm afraid. If others want to take it up I hope I can be of assistance and give some guidance but that's about all I can afford.

Sorry I don't have more positive news

Hey, thank you for looking into the matter. I managed to cobble together a script that uses ffmpeg to convert the streams into something acceptable for handbrake to use, so I'm OK -- but it's nevertheless a good bit of info to have about SVCDs and Handbrake dev.

awk · Post by **awk** » Thu May 17, 2007 6:04 pm

I've posted an updated (near final) patch to my website :

http://awkward.org/handbrake/mpeg2-ts-patch.tar.gz

There's one bug still in this patch - Transport Streams with anything other than AC-3/A.52 audio won't work correctly (you'll get silence) - unfortunately I don't have much test media to work on this problem with, all my transport streams have AC-3 audio in them. The streams I do have with MPEG audio are Program Streams (which work fine

jbrjake · Post by **jbrjake** » Fri May 18, 2007 5:09 pm

awk, so, for this and your mp2 audio patch -- do you think one or both are ready to commit? And I forget -- do you have an svn account yet to commit them yourselves, or would you prefer if one of us did it?

I'd like to get this stuff into the repository soon =)

awk · Post by **awk** » Sun May 20, 2007 11:56 pm

The patch has been updated for SVN Rev 591 (notably maurj's DTS changes).

I'm interested in anyone who has a Windows development system in giving my patch a try (note you'll need to update the makefiles etc. to add libhb/stream.c - no other changes should be neccessary).

I'm working on getting my linux system back to the state where it can build things (it's 'atrophied'

Once the transport stream patches at least build on a linux machine (and hopefully Windows) I'll commit them.

The MPEG-Audio fixes (Ticket #51) have already been submitted.

alynpier · Post by **alynpier** » Mon May 21, 2007 6:18 am

Hi,
I am not a developper, only a "single" user of Handbrake.
How can I add this feature to "my" copy of Handbrake ?
If someone can do this, is it possible to let it (the whole app) download somewhere ?

Thanks,
Hace a nice day,

fullerflyer · Post by **fullerflyer** » Tue May 22, 2007 4:37 am

alynpier wrote:Hi,
I am not a developper, only a "single" user of Handbrake.
How can I add this feature to "my" copy of Handbrake ?
If someone can do this, is it possible to let it (the whole app) download somewhere ?

Thanks,
Hace a nice day,

http://handbrake.m0k.org/trac/wiki/CompileGuide

...someone else was kind to point me to this a while back. I hope this is what you're looking for (the method/means of getting the most recent build under development).

Au Revoir!

alynpier · Post by **alynpier** » Tue May 22, 2007 12:43 pm

Thank you for that, I knew it.
I used it to "make" my own of HB 0.8.5b1, but I wondered how I could include awk's patch.

I think I have made it !!!! Myself, I can't believe it !

I just added awk's "stream.c" to "~/HB-dev/libhb" folder and then re-make the app.

I got a new Handbrake, I'll do some tests with mpeg files exported from EyeTV. These are TV programs in DVB-T.

alynpier · Post by **alynpier** » Tue May 22, 2007 12:48 pm

How can I import a mpeg file in HB ?
I just have the choice between a physical DVD, a DVD image or a VIDEO_TS folder...

If HB is able to convert from a mpeg file, how can we read it ?

HandBrake

MPEG-2 Transport Stream Support

Transport Stream support patch available

Re: Transport Stream support patch available

Yay!

Patch updated