2-Pass Encoding Optimization

[araftis]

While recently experimenting with different encoding rates and codecs, I noticed something when encoding a DVD. On my machine, the MP4 encoder is basically bound by the speed of the DVD drive. I found that if I copied the DVD to my HD prior to encoding, I got about 20 more FPS with MP4 encoding. The downside of this, of course, is that it doesn't save time, since I have to copy the movie to the HD first, which nullifies any gains in encoding speed.

So the idea is that during the first pass encoding, the DVD is copied to the HD as it's encoded. Hopefully, this'll take around the same time as just copying the DVD (on a fast enough machine). Then during the second pass, the movie data can be read from the HD, getting the boost in encoding speed. I'm not 100% sure how efficient this will be, unfortunately, since trying to write the DVD data and the encoded data might max out the HD transfer rates.

Also, at first, this may only help a handful of people, but with machines like Apple's Quad Core Mac Pros becoming more common, I think this could help quite a few people.

[dynaflash]

Well, here is food for thought. HB is not the best ripper when it comes to the newer dvd's (read: last two years). In fact, other than ignoring Css, HB does nothing with newer advanced copy protection whatsoever. So, in most cases, so to be sure, we recommend ripping the dvd first with a tool like MacTheRipper (in fact, the windows version requires that you use a ripper) anyways.

As more and more dvd's have newer advanced copy protection, the need for a third party ripper will just become greater, to the extent that we have removed "Ripper" from HB's description and in fact, have changed the "Rip" button to "Start" in the mac Gui.

So, a process like you describe becomes less and less beneficial as time goes on.

[araftis]

While I generally agree with your less is more approach in HandBrake, and that HandBrake should not try to be everything, I think this case may be an exception. Consider that it seems to take me, on average, about 30 minutes to copy a DVD to the hard drive. Assuming we're talking an average length film here, we're looking at (for me, on my system) about 40 minutes for each pass of the film encoding (using h.264, MP4 is faster). That means you're increasing the encoding time by nearly 37%, which to me, makes a good argument for HandBrake also needing to be a ripper as well as an encoder.

Special note here: Under the last version of HandBrake, I got 60 fps rather than just 40 fps with h.264 encoding, but I'm not sure if that drop is due to a regression, better encoding, or debug code.

Anyways, to put things into perspective, once you guys can handle subtitles better (my wife and I really need support for on/off subtitles, even if they're just written to a separate file I can mux back into the MP4 file myself), I plan to rip about 800 DVDs to hard drive. Adding 30 minutes to each rip increases my total time spent to encoding my library by 400 hours.

Also, on top of the time increase, needing to use two programs makes automation a lot more difficult process. For example, with iTunes, I insert a disc, it rips, and then ejects the disc, making it very easy to rip a large library of CD's. Eventually, it'd be cool if HandBrake could make it just as easy to rip my DVDs as iTunes does for my CDs.

So, that being said, shouldn't libdvdread be handling the read issues for ripping? It's the library used by MacTheRipper, so if that can handle the newer DRM systems found on DVDs, I would think HandBrake could, too.

[realityking]

I currently rip DVDs while others are Encoding, with a Harddrive this large enough is this quite possible. Maybe this could be a solution for you.

[araftis]

Well, that might work in the short term. I plan on buying two 1 TB drives to store our collection. I'm estimating needing about 1.5 TB to store everything at the quality I want, so HD space won't be an issue.

Currently, I'm not really ripping very much. I'm pretty much waiting for four features to be implemented, two of which have already been discussed in other threads (better subtitle support and remembering settings between sessions) and two I'll likely be opening other threads to discuss over the next week or so. Given some free time, I may even jump in and try to help code some of this work.

Just FYI, my ideal usage of HandBrake would be the following:

• 1. Set up ripping parameters once. This would include basic video settings like bit rate and how to treat various audio tracks. For example, always treat 5.1 audio as 6 channel discrete.
  2. Insert disc. HB, if not launched, would launch, and rip the disc, using my settings, to a predetermined folder.
  3. If needed, HB would sound something like an audio alert indicating it needed attention. For example, it might have problems detecting the main title on the disc, or it might not be able to select the primary audio track since your default language wasn't found.
  4. When done ripping, HB would eject the disc and wait for the next.

This really helps when working with a large number of titles. You don't really have to think about what you're doing expect during initial setup. Then, all you have to do is wait for the DVD tray to eject and swaps discs. All in all, it requires very little thought. At this point, this is mostly just food for though as to why I think HB should include the ripping as well as transcoding support.

Of course I might be able to just copy DVD's all day (say ten or so) and then allow HB to work on them in sequence over night, since you do have the queue feature.

Also, are you sure HandBrake isn't handling some of the newer copy protections? I was under the impression that Pirates of the Caribbean used one of the new schemes, and I ripped it with no problems. It would seem that as long as you guys keep up with the latest version of libdvdread (and maybe move to libdvdnav from libdvdplay, which you'll probably need to do anyways), then HandBrake shouldn't really have issues with reading most discs. After all, MacTheRipper appears to use libdvdread, too.

[jbrjake]

Also, are you sure HandBrake isn't handling some of the newer copy protections? I was under the impression that Pirates of the Caribbean used one of the new schemes, and I ripped it with no problems. It would seem that as long as you guys keep up with the latest version of libdvdread (and maybe move to libdvdnav from libdvdplay, which you'll probably need to do anyways), then HandBrake shouldn't really have issues with reading most discs. After all, MacTheRipper appears to use libdvdread, too.

Yes, but MTR is not open source, now is it? So how, pray tell, are we supposed to get all their custom modifications to handle new encryption schemes?

Saying we just need to keep up with the latest version of libdvdread is a morbid joke, right?

Please suggest a serious way of accomplishing what you ask HB to support.

[araftis]

Yes, but MTR is not open source, now is it? So how, pray tell, are we supposed to get all their custom modifications to handle new encryption schemes?

Well, you could just ask. After all, libdvdread is distributed under the GPL, not the LGPL. That means in order to continue using libdvdread, MTR pretty much has to make all changes available. In fact, it technically means they have to make the source code for their entire application available. Besides, given the complimentary features of your applications, it would seem like MTR would be willing to work with you.

On top of that, my understanding of the new "DRM" schemes on DVDs do not involve "new" encryption. Technically, they're not even DRM in the traditional sense. Makers of DVDs are limited by the DVD specification itself. They cannot introduce anything new into the specification, because this will break actual DVD playback hardware. If they do, they can't actually claim their software is a DVD, and yes, this has happened at least once.

They can, however, bank on areas of vagueness in the specification. For example, the ARccOS doesn't actually introduce anything new onto the DVD. It is a fully compliant DVD. On the other hand, much of the software for reading DVDs is not actually compliant with the specification. ARccOS depended on this with their scheme. By introducing "bad" blocks into the bit stream, they throw off players/readers that don't fully conform to the DVD specification. Of course this often backfires, because not all playback hardware fully conforms, either, which is what recently bit Disney in the butt.

So what's the point? It's that I think it's completely reasonable to expect the underlying libraries used by HB, VLC, MTR, and many others will be updated and maintained as the libraries grow to fully support the DVD specification. Sadly, they really can't be fully compliant out of the box, since they have to implement mostly through reverse engineering, given the exorbitant price of obtaining official the DVD specification.

Saying we just need to keep up with the latest version of libdvdread is a morbid joke, right?

Well, I can't specifically answer this one in the case of libdvdread. However, as part of my job, I do develop software libraries professionally, and as such, I attempt to make moves from one version to the next as painless as possible. As such, I assumed keeping up with the latest version would be fairly painless. This, of course, could be an incorrect assumption in the case of libdvdread. Still, as previously stated, libdvdread is licensed under the GPL, so in the very least, all changes made by third parties, including MTR, should be publicly available.

Please suggest a serious way of accomplishing what you ask HB to support.

I thought this was a perfectly reasonable suggestion and not a morbid joke. As a software developer, I'm expected to keep my active code somewhat up-to-date with the latest versions of libraries. Sure, for testing and stability purposes we might lag a minor version or two behind, but rarely do we fall too far behind.

Is libdvdread really so badly written that you're stuck on an older version and unable to move forward? This may be the case, but it would seem unlikely, but sad if true.

As a final note, I know that the move from libdvdplay to libdvdnav will probably be a much larger, more painful move. I mostly mentioned it in my previous post because VLC, the maintainers of libdvdplay, have deprecated the library in favor of libdvdnav, so one way or the other, libdvdplay is likely to stagnate.

[s55]

Is libdvdread really so badly written that you're stuck on an older version and unable to move forward? This may be the case, but it would seem unlikely, but sad if true.

We already use the latest version. The latest version is around 2 years old +
It hasn't been maintained in a long time.

Afaik MTR has not made its source available despite the GPL so we're stuffed there. Infact you have to Pay to get the latest version of MTR :/

I'm not sure there are any developers around that would have the knowledge and/or time to move away from libdvdread

[jbrjake]

Well, you could just ask. After all, libdvdread is distributed under the GPL, not the LGPL. That means in order to continue using libdvdread, MTR pretty much has to make all changes available. In fact, it technically means they have to make the source code for their entire application available.

Yes, but they *don't*. Unless you want to sue them, I don't see how we're going to get that code. They have a strong monetary incentive not to share it. After all, if we fold in their modifications, then all of a sudden we won't be telling people to go pay them $30...

On top of that, my understanding of the new "DRM" schemes on DVDs do not involve "new" encryption.

Okay, yes, I misspoke. Technically they are inserting junk data and not encrypting. But considering that it means you need to use a special, secret method of accessing the data or it's unintelligible, it's effectively a form of encryption.

So what's the point? It's that I think it's completely reasonable to expect the underlying libraries used by HB, VLC, MTR, and many others will be updated and maintained as the libraries grow to fully support the DVD specification.

They *do* fully support the DVD specification. But as far as I know there are exactly zero open source projects that handle DVDs that intentional *break* spec.

Please suggest a serious way of accomplishing what you ask HB to support.

I thought this was a perfectly reasonable suggestion and not a morbid joke. As a software developer, I'm expected to keep my active code somewhat up-to-date with the latest versions of libraries. Sure, for testing and stability purposes we might lag a minor version or two behind, but rarely do we fall too far behind.

Err...what?

Kind of hard to update when libdvdread hasn't been touched in over half a year.

As a final note, I know that the move from libdvdplay to libdvdnav will probably be a much larger, more painful move.

Where do you see libdvdplay in our code?

[araftis]

Is libdvdread really so badly written that you're stuck on an older version and unable to move forward? This may be the case, but it would seem unlikely, but sad if true.

We already use the latest version. The latest version is around 2 years old +
It hasn't been maintained in a long time.

Actually, HB is a couple minor revs out of date. I just checked out and built the code. HB appears to be on version 0.9.4, while the latest is 0.9.7. Unfortunately, the NEWS file in the libdvdread source doesn't give what changes were made between 0.9.4 and 0.9.7. However, if you can point me at a list of movies that are known to fail in HB due to "copyright" incompatibilities, I can try linking HB against the latest version and see if it makes any difference.

Afaik MTR has not made its source available despite the GPL so we're stuffed there. Infact you have to Pay to get the latest version of MTR :/

Sounds like someone needs a cease and desist order. MTR was apparently breaking the terms of the GPL before, and if they're going to start charging, then they're not only breaking the terms of the GPL but the spirit of the GPL as well.

I'm not sure there are any developers around that would have the knowledge and/or time to move away from libdvdread

Sadly, that's not me, either. If there's a better alternative to libdvdread, I could possibly look at helping move the code to a new library, but I don't have the expertise to support libdvdread itself.

[jbrjake]

Actually, HB is a couple minor revs out of date. I just checked out and built the code. HB appears to be on version 0.9.4, while the latest is 0.9.7.

Um, no.

How did you reach that conclusion if you've checked out the code? contrib/version_libdvdread.txt clearly states that HB uses libdvdread 0.9.7 and the modification date on the source files themselves matches that.

[araftis]

Well, you could just ask. After all, libdvdread is distributed under the GPL, not the LGPL. That means in order to continue using libdvdread, MTR pretty much has to make all changes available. In fact, it technically means they have to make the source code for their entire application available.

Yes, but they *don't*. Unless you want to sue them, I don't see how we're going to get that code. They have a strong monetary incentive not to share it. After all, if we fold in their modifications, then all of a sudden we won't be telling people to go pay them $30...

Well, in all likely hood, they're asking for a world of hurt. Every single company that I know that has tried to charge money for a DVD Ripper has been issued a cease and desist order from the CCA. And they have the lawyers to hurt people. You can get away with distributing "free" software, especially if it originates outside the U.S., but as soon as you turn the download into a financial transaction, you're asking for trouble.

On top of that, my understanding of the new "DRM" schemes on DVDs do not involve "new" encryption.

Okay, yes, I misspoke. Technically they are inserting junk data and not encrypting. But considering that it means you need to use a special, secret method of accessing the data or it's unintelligible, it's effectively a form of encryption.

I was just saying it shouldn't be worse than the original reverse engineering that produced libdvdread. My understanding, and this may not be 100% correct, is that what they did was insert a bad packet. However, if you're following the spec completely, your software/firmware will never read that packet, because it was told to "skip" it. The readers that broke were just reading the stream, assuming all packets would be valid.

Please suggest a serious way of accomplishing what you ask HB to support.

I thought this was a perfectly reasonable suggestion and not a morbid joke. As a software developer, I'm expected to keep my active code somewhat up-to-date with the latest versions of libraries. Sure, for testing and stability purposes we might lag a minor version or two behind, but rarely do we fall too far behind.

Err...what?

Yeah, that didn't come out too well. Basically, you guys are doing what you should be doing. HB shouldn't ever be a full DVD ripper, especially since you don't need to worry about menus and what not. It was mostly just my understanding that if you continued to update libdvdreader, your reading code would update along the way, and you'd be able to handle new attempts at copy protection.

Kind of hard to update when libdvdread hasn't been touched in over half a year.

True. I had thought it was under more active development, especially since we've been seeing regular updates of the programs that make use of it.

As a final note, I know that the move from libdvdplay to libdvdnav will probably be a much larger, more painful move.

Where do you see libdvdplay in our code?

You're right, I don't. I made the bad assumption, prior to checking out the code about 1/2 hour ago, that you would use libdvdplay to handle things like multiple story branches and angles. Perhaps in the future, but then that'll make using libdvdnav a lot easier

[araftis]

Actually, HB is a couple minor revs out of date. I just checked out and built the code. HB appears to be on version 0.9.4, while the latest is 0.9.7.

Um, no.

How did you reach that conclusion if you've checked out the code? contrib/version_libdvdread.txt clearly states that HB uses libdvdread 0.9.7 and the modification date on the source files themselves matches that.

Actually, I looked in the header file in contrib/include/dvdread/dvd_reader.h. Taking a second look, I see that they're listing two versions, but since one's a comment and the other is the actual #define statement, and the #define statement shows version 0.9.7, I'll take it that you're on the most recent version. Sorry about confusing that.