Page 1 of 8

Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Tue Feb 03, 2009 1:44 pm
by nightstrm
As more and more of us are starting to encode from Bluray and HDDVD sources, I figured I'd start a thread so we can discuss our strategies/workflows for getting additional information off of the discs and into our encodes. As this is not Handbrake specific, I've put this thread in the Tiki Bar forum. If anyone has any information to add or discuss, please add it to this thread.

My current workflow is:
1.) Rip the disc using AnyDVD HD (Windows).
2.) Use eac3to (Windows) to extract separate audio, video, chapter, and subtitle files. For Bluray, this may only be necessary to grab chapter file if feature is contained in a single .m2ts file and has a TrueHD soundtrack (DTS will need to be converted to AC3 using eac3to).
3.) Use tsMuxeR (Windows) to remux the audio and video into .m2ts.
4.) Use ChapterGrabber (Windows) to convert chapter timing from 24fps to 23.976fps (only HDDVD VC1 sources).
5.) Use suprip (Windows) to convert subtitle track into SRT file (via OCR... lots of hand editing required; can also try to find online).
6.) Use Handbrake (Mac) to encode two videos from .m2ts file, using my slightly modified Universal and AppleTV presets. (preset details here: http://forum.handbrake.fr/viewtopic.php ... 755#p50133)
7.) Use SubCleaner (Mac) to cleanup SRT for Apple device compatibility.
8.) Use Subler (Mac) to add chapter and subtitle tracks, set HD/SD tags (hdvd, stik, cnID), and other metadata.
9.) Import encoded files into iTunes.

Current objectives of mine:
1.) Get chapter markers into encodes -- RESOLVED -- Currently investigating using mp4chaps to insert chapters based on information given in the title.chapters.txt file generated by eac3to (after modifying the file to use syntax expected by mp4chaps). So far, I've only tested with an HDDVD source and the resulting chapter markers get progressively further off as the movie goes on (ex. last chapter of current HDDVD encode is approximately eight seconds early). Have not attempted on a Bluray-sourced encode yet.

2.) Get soft subtitles into encodes -- RESOLVED -- Currently working after converting .sup file generated by eac3to with suprip and adding to .m4v file using Subler. This involves a PITA OCR process that creates a lot of mistakes, which need to be manually fixed. Noticed that iTunes/Quicktime appear to have a problem displaying 3-line subtitles, whereas the AppleTV does not (it combines lines 2 and 3, at least on the source I have been testing with).

3.) Get forced subtitles into encodes -- RESOLVED -- See http://forum.handbrake.fr/viewtopic.php?f=14&t=9258 for information on generating VOBSUB from BR PGS files and feeding into Handbrake using MKV container.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Wed Feb 04, 2009 2:13 am
by refulgentis
Definitely use SubCleaner (http://mac.softpedia.com/get/Word-Proce ... aner.shtml) on your SRT before using Subler. It is intended for use with the AppleTV, and does an excellent job of making my SRT files look fine in Quicktime.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Wed Feb 04, 2009 1:35 pm
by nightstrm
OK, SubCleaner is awesome... lets you limit the number of lines (necessary for iTunes/QT playback), and strip problematic characters for the AppleTV. So it is now part of my workflow as well.

I ripped "X-Files: Fight the Future" Bluray last night and am happy to report that the chapter markers appear to be in-sync with the video. I'm thinking the HDDVD chapter markers are off due to the VC1 pulldown removal and the chapters being synced at 29.97 instead of 23.976. Make sense? Anyone have an easy way to convert timecodes from 29.97 to 23.976? My next test is to try a AVC-encoded HDDVD to see if the chapter problem is limited to only those with VC1 sources. (I was close, it's due to the chapter markers being timed @ 24fps, while the video is @ 23.976fps. ChapterGrabber will take care of this and can even grab chapter names from Internet if not present in chapter file.)

At this point, the only remaining issue for Bluray sources is forced subtitles. I need to find a way to either make a QT-compatible subtitle track marked as forced (so it always shows), or figure out a way to get a SRT into Handbrake so it can hard code them in the picture, similar to the forced subtitle option for DVDs.

As for HDDVDs, I still need to figure out how to adjust the chapters to sync correctly to the encoded video (hopefully this is limited to VC1 sources only), and how to get forced subtitles to show in the encoded file.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 12:57 am
by nightstrm
I've made a lot of progress in 24 hours, and now have encoded both a Bluray and HDDVD title and successfully inserted chapter markers and soft subtitles. To fix my problem with chapter markers getting progressively further off on the HDDVD I ripped, I used a freeware program named ChapterGrabber (Windows) to convert the chapters from using 24fps to 23.976fps to calculate the timing. Once completed, the chapters occur in their correct spots throughout the movie.

At this point, the only thing left to do is figure out a way to encode forced subtitles in these encodes. I've found one of my Bluray discs, Iron Man, that uses in a couple foreign dialog scenes. I've been able to extract them to their own SRT file, but I have not found a way to either insert them into Handbrake as a forced subtitle, or mark the subtitle track inserted by Subler as forced in any Apple-compatible fashion. I'll continue looking, but if anyone else has any thoughts, please share.

I can't believe my workflow now has 12 steps in it... and relies heavily on Windows (via VMWare Fusion).

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 6:39 am
by ssj2_goha
Can you post links to some of the free software?
I'm looking into doing this.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 1:47 pm
by nightstrm
ssj2_goha wrote:Can you post links to some of the free software?
I'm looking into doing this.
It's all free, except for AnyDVD HD of course. You're lucky I don't feel like working this morning... otherwise I'd tell you to go Google it. :lol:

eac3to (Windows) - http://forum.doom9.org/showthread.php?t=125966
tsMuxeR (Windows) - http://forum.doom9.org/showthread.php?t=134104
ChapterGrabber (Windows) - http://jvance.com/pages/ChapterGrabber.xhtml
suprip (Windows) - http://exar.ch/suprip/
HandBrake (Mac) - http://www.handbrake.fr/
mp4tags (Mac) - http://code.google.com/p/mp4v2/ (part of mp4v2 project, need to build from SVN)
mp4chaps (Mac) - http://code.google.com/p/mp4v2/ (part of mp4v2 project, need to build from SVN)
MetaX (Mac) - http://www.kerstetter.net/page53/page54/page54.html
SubCleaner (Mac) - http://mac.softpedia.com/get/Word-Proce ... aner.shtml
Subler (Mac) - http://code.google.com/p/subler/

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 2:12 pm
by Starhawk
Wow, fantastic thread, nightstrm! Im not even encoding any Blu-rays right now, but it's good to know this is here for when I do (which will probably now be sooner rather than later).

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 2:19 pm
by Northy
Get guide mate :D

I have couple of questions:
mp4chaps ~ Can you add chapters to MP4's (M4V) that are already tagged?
SubCleaner ~ Is that like Jubler where you can clean up the subs and convert to SRT?
Subler ~ Is it better than Muxo?

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 2:34 pm
by nightstrm
Northy wrote:mp4chaps ~ Can you add chapters to MP4's (M4V) that are already tagged?
Yes, you can even add chapters to iTunes-purchased content that doesn't have them (for example, the Peanuts Holiday shows have 2-3 episodes in each file, but no chapters to separate them).
SubCleaner ~ Is that like Jubler where you can clean up the subs and convert to SRT?
To be honest, I haven't used Jubler and there isn't much info out there as to exactly what SubCleaner does. The biggest advantage to using it that I've found is that it will take 3-line subtitle blocks and split them up so they will all show in Quicktime/iTunes. You can also have it strip characters that have been problematic on the AppleTV.
Subler ~ Is it better than Muxo?
Subler is a recent development by Rituska, a Handbrake developer. Initial reactions that I've seen indicate it has some features not yet available via Muxo. You can read more info about it here: http://forum.handbrake.fr/viewtopic.php ... 97&start=0

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 2:37 pm
by Northy
Thanks for the answers, SubCleaner is not for me, ruined my English subs which had German characters in them (Names mainly), the other 2 look good so far :D (I just hope I can add subs that aren't the length of the film (for example what should be forced subs, Muxo had an issue with them :evil: )

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 2:51 pm
by nightstrm
Northy wrote:Thanks for the answers, SubCleaner is not for me, ruined my English subs which had German characters in them (Names mainly), the other 2 look good so far :D (I just hope I can add subs that aren't the length of the film (for example what should be forced subs, Muxo had an issue with them :evil: )
Haven't tried short sub tracks yet... since I can't figure out a way to actually force them to be shown on startup.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Thu Feb 05, 2009 4:45 pm
by nightstrm
As of SVN revision 265 of the mp4v2 library, the information in this post is no longer necessary -- mp4chaps can take the output of eac3to (or ChapterGrabber if you needed to convert 24->23.976) and parse the information correctly. Thanks to the mp4v2 team for responding to my request. :)

It has been brought to my attention that I failed to mention how to get the output from ChapterGrabber to work with mp4chaps.

The output from eac3to (and ChapterGrabber) will look like this:
CHAPTER01=00:00:00.000
CHAPTER01NAME=Living Weapon
CHAPTER02=00:05:00.750
CHAPTER02NAME=A Better World
CHAPTER03=00:09:31.937
CHAPTER03NAME=Aboard Serenity (Main Titles)
CHAPTER04=00:14:49.088
CHAPTER04NAME=Going for a Ride
...
You need to edit the file to look like this:
00:00:00.000 Living Weapon
00:05:00.750 A Better World
00:09:31.937 Aboard Serenity (Main Titles)
00:14:49.088 Going for a Ride
...

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Fri Feb 06, 2009 6:53 pm
by refulgentis
Northy wrote:Thanks for the answers, SubCleaner is not for me, ruined my English subs which had German characters in them (Names mainly), the other 2 look good so far :D (I just hope I can add subs that aren't the length of the film (for example what should be forced subs, Muxo had an issue with them :evil: )
Change preferences of SubCleaner to not strip accents -- that should fix it.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sat Feb 07, 2009 6:58 pm
by rICh
Thank you nightstrm for the great info.

I was wondering if you would be so kind to let me know the h.264 advanced options you are using in handbrake for HD content? I've used a your advanced settings with DVD sources to fantastic effect. :D

Thank you for your time, patience and knowledge.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sat Feb 07, 2009 8:09 pm
by iJoel
nightstrm wrote:It has been brought to my attention that I failed to mention how to get the output from ChapterGrabber to work with mp4chaps.

The output from eac3to (and ChapterGrabber) will look like this:

I can make a converter for you if you think that'd be useful.
Would be trivial to make, and I'm sure you'd find it useful if
your doing all of that manually.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sat Feb 07, 2009 8:10 pm
by nightstrm
rICh wrote:Thank you nightstrm for the great info.

I was wondering if you would be so kind to let me know the h.264 advanced options you are using in handbrake for HD content? I've used a your advanced settings with DVD sources to fantastic effect. :D

Thank you for your time, patience and knowledge.
I make two copies, using the presets below:
HD - AppleTV preset @ 57%, detelecine, decomb, no chapters
SD - Universal preset @ 57%, detelecine, decomb, only AAC soundtrack (no AC3), no chapters

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sat Feb 07, 2009 8:11 pm
by nightstrm
iJoel wrote:
nightstrm wrote:It has been brought to my attention that I failed to mention how to get the output from ChapterGrabber to work with mp4chaps.

The output from eac3to (and ChapterGrabber) will look like this:

I can make a converter for you if you think that'd be useful.
Would be trivial to make, and I'm sure you'd find it useful if
your doing all of that manually.
I actually was hoping to get around to writing a converter myself, but wanted to see if I could get the mp4v2 crew to simply accept it as an input. :)

If you'd like to put something together, it would be appreciated.

EDIT: Request submitted to mp4v2 team.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Mon Feb 09, 2009 1:16 pm
by nightstrm
I've tested a patch to mp4chaps that allows direct reading of the chapter files from eac3to. It works as it should; I expect it will be committed to SVN as soon as the dev gets around to it. Once it makes it into SVN code, I'll update my first post with the new information.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Mon Feb 09, 2009 2:28 pm
by Northy
refulgentis wrote:
Northy wrote:Thanks for the answers, SubCleaner is not for me, ruined my English subs which had German characters in them (Names mainly), the other 2 look good so far :D (I just hope I can add subs that aren't the length of the film (for example what should be forced subs, Muxo had an issue with them :evil: )
Change preferences of SubCleaner to not strip accents -- that should fix it.
I'll try it thanks :D

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Fri Feb 13, 2009 2:11 pm
by nightstrm
Just a little update on forced subtitles...

I've found that if I add both the forced and non-forced subtitle track using Subler, I can select the forced subtitle track when I start playing the movie via the AppleTV. On subsequent viewings, the subtitle track will remain in the enabled state and display correctly. While this "works", I'm still investigating a way to cut out the manual step of selecting the track in the first place.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sat Feb 14, 2009 7:05 pm
by iJoel
nightstrm wrote:Just a little update on forced subtitles...

I've found that if I add both the forced and non-forced subtitle track using Subler, I can select the forced subtitle track when I start playing the movie via the AppleTV. On subsequent viewings, the subtitle track will remain in the enabled state and display correctly. While this "works", I'm still investigating a way to cut out the manual step of selecting the track in the first place.
I'll look into this for you when I'm testing the audio/subtitle capabilities of the ATV.
But, really, aside from burning the subs directly onto the encode -somehow- (which I agree, is
most definitely not ideal, and defeats the purpose of forced subs), I can't really
see how the original choosing of the forced subs could be avoided. But I'll look
into the more advanced settings and see if the ATV checks
any flags when it originally plays a video, and *hopefully* I'll
come up with a solution for you. Cause I agree, that would make HD
encoding perfect ^__^.

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sun Feb 15, 2009 6:04 pm
by mac.jedi
nightstrm wrote:I've tested a patch to mp4chaps that allows direct reading of the chapter files from eac3to. It works as it should; I expect it will be committed to SVN as soon as the dev gets around to it. Once it makes it into SVN code, I'll update my first post with the new information.
Hi nightstrm! Thanks for the info! Any possibility you could direct me to the mp4chaps patch?

Also just to throw it out there, I'd like to find a way to automate eac3to to extract the chapters, audio and video (cause I'm a lazy [Censored]). Of course, the problem is that the selections differ from disk to disk. I've got a script that I've been using to extract the chapters from all the BD's in a batch folder. Since chapters are always the first selection this works out great, but some BD's contain multiple video and some do not, so I can't always predict if item 3 or 4 is going to be the main audio. Any help would be greatly appreciated.

Also, I've found that if you save the output from chapter grabber to: Save as type: Timecodes Text File (*.txt), the chapters output in the correct format, but without any chapter titles. I don't really care about the titles, just as long as I can skip through the movie.

Anyway, thanks for your help.

Here are the scripts I've been using if anyone is interested. Right now, it searches the batch folder for a tsMuxeR meta file to find the BD to process and extract the chapters. I then have to open the file in chapter grabber to save in the "Timecodes Text File" format. If anyone knows of a command line version that will do this, that would be awesome.

Thanks Again!

Code: Select all

:: run_batcheac3to.cmd UPDATED: 02-08-2009

@echo off

:: set your input directory
set batchfolder="Z:\SpeedDisk\Batch Rip"

cd /d %batchfolder%

:: set the search string appended to the filename of the meta file ex. "*_BD.meta"
for /r %%i in (*_BD.meta) do set metafile="%%i" & call batch_eac3to.cmd

:: endscript

Code: Select all

:: batch_eac3to.cmd UPDATED: 02-08-2009

for /f "delims=." %%n in ("%metafile%") do set infile="%%~dpn"
for /f "delims=." %%n in ("%metafile%") do set outfile="%%~dpnn.chapters.txt"
"C:\Users\Robert Yamada\Desktop\eac3to\eac3to.exe" %infile% 1) 1: %outfile%
echo "  - - - - - - - - - - - - - - - -"
)
:: endscript

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sun Feb 15, 2009 7:28 pm
by mac.jedi
nightstrm wrote: 7.) Use mp4tags (Mac) to set hdvd, stik, and cnID on both HD/SD files (for iTunes grouping to behave).
How do the tags need to be set for combining the HD/SD files (especially setting the cnID's). Would you mind explaining this process in more detail? If possible, I'd like to see the command line syntax as well.

Thanks again!

UPDATE: found nightstrm's post on macrumors. all is well. thanks.
http://forums.macrumors.com/showpost.ph ... stcount=11

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Sun Feb 15, 2009 11:55 pm
by nightstrm
Hi nightstrm! Thanks for the info! Any possibility you could direct me to the mp4chaps patch?
It has been checked in as part of rev 264 (and fixed in 265). :)

Re: Advanced Bluray/HDDVD Workflow & Features Discussion

Posted: Mon Feb 16, 2009 5:34 am
by peterjcat
nightstrm wrote:I've made a lot of progress in 24 hours, and now have encoded both a Bluray and HDDVD title and successfully inserted chapter markers and soft subtitles. To fix my problem with chapter markers getting progressively further off on the HDDVD I ripped, I used a freeware program named ChapterGrabber (Windows) to convert the subtitles from using 24fps to 23.976fps to calculate the timing. Once completed, the chapters occur in their correct spots throughout the movie.
Thanks for sharing your workflow, nightstrm. I've now tweaked mine so it's, er, exactly the same :)

Are you finding that most HDDVDs have subtitles coded for 24.000fps or is it the exception? Is there a way to tell apart from trial and error? And is it the same for Blu-ray?