Upgrade of MKV collection from H.264 to H.265 possible?

General questions or discussion about HandBrake, Video and/or audio transcoding, trends etc.
Post Reply
FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Fri Aug 18, 2017 5:51 am

Hi everybody,

whoever uses software like handbrake knows that it is a really time consuming task to work yourself through your DVD and Bluray shelf and free your content from its possibly mortal remains.

I did this for a huge collection in the last years and used the codec h.264 for this task. Know, since H.265 is becoming more widespread and popular and devices like the Odroid C2 allows to play them over Kodi, I somehow have the feeling that I did the job unfortunately one year too early.

Of course, it is not direct necessarily to have everything in h.265 since it is mainly a matter of size. But for a big collection this sums up.

So long story short: Is there a handy and convenient way to reencode the mkv files with the newer codec as a background task to end up with identical quality as compared to again putting every single disk back in the drive?

User avatar
Ritsuka
HandBrake Team
Posts: 1083
Joined: Fri Jan 12, 2007 11:29 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by Ritsuka » Fri Aug 18, 2017 5:59 am

No.

User avatar
BradleyS
Moderator
Posts: 1729
Joined: Thu Aug 09, 2007 12:16 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by BradleyS » Fri Aug 18, 2017 11:54 am

Generally speaking, you will want to go back to your original sources to achieve the best quality and compression.

That said, I think it's a bit early to be switching to H.265 en masse (or at least for me). H.264 isn't going away. Better to enjoy your current collection, and maybe experiment with some H.265 encoding with newer encodes if you're curious.

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Fri Aug 18, 2017 6:24 pm

So you think H.265 is still to experimental? Since it's used in Germany for DVB-T2 broadcasting I thought it's already rock stable.

And of course you are right. They did not turn bad over night. But when I see that I can shrink the used storage to a half with the new codec I get really excited. Espacially since I have some Bluray Seasons here, which are extreme in size, because of some grainy effects, like Walking Dead. But maybe I have to experiment more with noise reduction.

User avatar
BradleyS
Moderator
Posts: 1729
Joined: Thu Aug 09, 2007 12:16 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by BradleyS » Fri Aug 18, 2017 6:35 pm

It's not that H.265 is experimental—the spec is stable. But current encoders haven't had any major breakthrough in compression efficiency, at least not for high/archival quality. x265 can make low bit rate video look surprisingly good, however. So even if more devices supported hardware accelerated H.265 decoding tomorrow, I'm not sure it's worth throwing electricity at quite yet.

On the other hand, x264 is mature and extremely well tuned at this point. Easy to get reasonably small, high quality encodes out of it and hardware decoding is basically everywhere.

NLMeans will certainly work some wonders on file size. For grainy sources, even more than switching to H.265.

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Fri Aug 18, 2017 6:48 pm

Hmm I found in an detailed guide some recommended settings for it, but anyway season 4 Walking Dead is still nearly 90GB with quality settings at 21.
This series is definitely on my list of things I want to give a second try to decode with different settings.

mduell
Veteran User
Posts: 6755
Joined: Sat Apr 21, 2007 8:54 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by mduell » Fri Aug 18, 2017 8:13 pm

What you should be doing, now (and the first time), is ripping the disks 1:1 with MakeMKV or similar; storage is cheap. This way you never have to touch the physical disk again.

Any encodes for size/compatibility/future codecs should be done from those rips.

Going from your already degraded H.264 encode to another H.265 encode will only make quality worse (perhaps not perceptibly so, depending on settings).

nhyone
Bright Spark User
Posts: 236
Joined: Fri Jul 24, 2015 4:13 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by nhyone » Sat Aug 19, 2017 3:39 am

After evaluating x265 for a while, I decided to continue using x264 until I have more compelling reasons to switch.

My reasons:
  1. x265 is very slow. I use a tweaked slower preset and it is much slower than x264's tweaked veryslow (can't remember how slow, except it was really slow)
  2. It took a long time for people to figure out settings for 1080p videos
  3. Need to use conservative settings as x265 has a tendency to smooth
Point 1: I looked at the x265 presets and feel that presets before slower all trade off significant quality for time. slower is the first one that turns on most of the x265 knobs. veryslow and placebo run "exponentially" slower, but have diminishing returns.

Point 2: be prepared to re-encode if you did it wrongly the first time, which you will inevitably do so. x265 is tuned for 4K. A different set of parameters is needed for 1080p. Sometimes it is due to x265 itself. For example, the new delta tables in 2.4.

Point 3: as a result, my encodes are only 20-30% smaller than a x264 encode, which is just marginally acceptable. No 50% savings for me.

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Sat Aug 19, 2017 6:12 am

mduell wrote:
Fri Aug 18, 2017 8:13 pm
What you should be doing, now (and the first time), is ripping the disks 1:1 with MakeMKV or similar; storage is cheap. This way you never have to touch the physical disk again.

Any encodes for size/compatibility/future codecs should be done from those rips.

Going from your already degraded H.264 encode to another H.265 encode will only make quality worse (perhaps not perceptibly so, depending on settings).
Storage is'nt that cheap. For sure thinking back when I had a ridiculous gigantic 10GB hard disk the storage is nowadays extremely cheap. But storing everything as untouched mkv and in addition as a compressed mkv is also an extreme big Folder.

I for example ripped my cds as flac, because there size really does not matter and to have them in principle archived is a nice feeling. But a single Bluray movie can be so big.

And why should I compress them then in a next step anyway? I just checked storage. The compressed collection is around 1.5TB. Considering that the raw files might be 4 times bigger then this would use the complete storage of my NAS. Because of that I rather stick with compressed files only and grab out every single disc again.

@nhyone:
This I did not know. I mean for new movies speed does not matter this much, but for reencoding from beginning it would be a big downfall. And I am also not an expert with settings. 30% is already something but it seems it really does not be worth the effort.

So maybe you convinced me that I should just stick to h.264. Is here a very good tutorial about optimized settings for different content? Maybe I can optimize my settings a bit. Especially captions caused me to rip some movies a second time. Because I usually like to have for all files, whenever available my mother tongue and English as selectable captions. But sometimes there are also in addition some captions for only few sentences when for example Arabic or Chinese is spoken. I never understood well how I get this forced flags but still have the complete subtitle selectable. But this is maybe a different story.

Woodstock
Veteran User
Posts: 3293
Joined: Tue Aug 27, 2013 6:39 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by Woodstock » Sat Aug 19, 2017 4:46 pm

I won't go into the esoterics of which settings do what, but I will champion the idea of starting from the original source if possible, when you do any encode. No matter what settings you use, a "child" encode will, at best, recreate the artifacts introduced in the first encode, while adding its own.

This is where a slight modification of mduell's suggestion can help - use another tool to move the video off the optical media as quickly as your drive can manage, while letting handbrake churn through the material at its own pace. You will temporarily use up a lot of disk space for the raw rips, but you don't have to sit around waiting to do disk changes as handbrake finishes each disk, having done all that before hand. It's quite easy to rip enough disks in 5 hours to keep handbrake busy for several days.

rollin_eng
Veteran User
Posts: 3190
Joined: Wed May 04, 2011 11:06 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by rollin_eng » Sun Aug 20, 2017 10:37 am

But storage IS cheap.

Its getting to the point where encoding just to save space is not worth it.

DougH
Posts: 18
Joined: Fri Sep 23, 2016 12:52 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by DougH » Sun Aug 20, 2017 12:51 pm

It might be "cheap" in the US but a single HDD - say a WD red 6tb lists for $290.00 (as of this morning) in Canada on Amazon. I could maybe find it cheaper if I did a little hunting but I just don't have 290 laying around.

rollin_eng
Veteran User
Posts: 3190
Joined: Wed May 04, 2011 11:06 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by rollin_eng » Sun Aug 20, 2017 1:04 pm

Obviously 'cheap' is always relative but encoding takes power which also costs money, add in any time you spend on it and it soon isn't always a good idea.

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Sun Aug 20, 2017 7:32 pm

Same here. I have a Synology 2Bay Nas which I recently upgraded to 2*6TB WD Red, giving me around 6TB usable space. That was nearly 500€, which is nearly 600$.

They are not the biggest HDDs but already quite big.

And again, when I have the hole collection stored as uncompressed mkv's, why should I start using Handbrake at all?

mduell
Veteran User
Posts: 6755
Joined: Sat Apr 21, 2007 8:54 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by mduell » Sun Aug 20, 2017 8:35 pm

You're inflating the costs with redundancy you don't need because you still have the original source disks if you were to lose that hard drive.
FordPrefect wrote:
Sun Aug 20, 2017 7:32 pm
And again, when I have the hole collection stored as uncompressed mkv's, why should I start using Handbrake at all?
If you don't need to, don't!

You might need to for compatibility (not every device supports mpeg2/avc/vc1 plus ac3/dts in mkv) or portability (storage constrained portable devices that need a one-off encode for playing on).

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Mon Aug 21, 2017 5:16 am

The redundancy is not important for the movies, but I have different stuff on this device, like all my personal data. And there I want to have the redundancy.
I mean if the size of your collection is small enough or your storage is huge enough to not compress your media it's fine. But I think I've hardly ever noticed some quality problems with the compressed stuff. So why shouldn't I compress it?
An uncompressed DVD is in the range of 5GB and uncompressed Bluray around 30-40 GB. That makes roughly 1.6TB for movies alone.
TV Series espacially Bluray ones are becoming just to big. A season alone might be end up with something like 200GB.

DrXenos
Bright Spark User
Posts: 209
Joined: Sat Mar 16, 2013 1:19 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by DrXenos » Sat Sep 02, 2017 1:43 pm

FordPrefect wrote:
Fri Aug 18, 2017 5:51 am
Hi everybody,

whoever uses software like handbrake knows that it is a really time consuming task to work yourself through your DVD and Bluray shelf and free your content from its possibly mortal remains.

I did this for a huge collection in the last years and used the codec h.264 for this task. Know, since H.265 is becoming more widespread and popular and devices like the Odroid C2 allows to play them over Kodi, I somehow have the feeling that I did the job unfortunately one year too early.

Of course, it is not direct necessarily to have everything in h.265 since it is mainly a matter of size. But for a big collection this sums up.

So long story short: Is there a handy and convenient way to reencode the mkv files with the newer codec as a background task to end up with identical quality as compared to again putting every single disk back in the drive?
I, too, have a huge collection. That's why, from day one, my goal was to retain all the information gleaned from the initial transcoding. If I wanted to re-transcoding them in a newer format, or just to fix an error, it is trivial to redo them.

I've actually gone through three iterations for attempts to keep the transcoding info.

1. Initially, I kept the HandBrake queue files (Windows GUI). This was problematic because the format changes a lot (not a criticism, it is what it is). After the first breaking change, I wrote a parser to convert all my scripts to the new format. I soon realized this was untenable. Plus, it doesn't contain information for all the other tools I use.

2. Next, I create a tool that used AutoIt to automate the GUI. With this, I could write AutoIt scripts for HandBrake and others tools to perform the transcoding. I wrote a tool to convert my HandBrake queue files to AutoIt scripts. This was OK, but very slow and a little flaky (as automating a GUI always it). Plus, I still had to update it as HandBrake changed (which could be time consuming).

3. What I currently have is I just wrote my own scripting language, specifically geared towards transcoding. It uses tools like HandBrake (command line version) and MakeMKV to analyze the source, and it "knows" what I want (e.g., the settings, what tracks, etc.). It takes options when I desire something different. It generates a batch file to do all the transcoding for a script. I use another batch file that will run multiple scripts and handle command to pause, shutdown, etc.

The language basically looks like a CSV file with variables. Most of the time, I just give it sources/titles and destination, but it had many options and features to deal with odd or special situations.

I like it because it does all the heavy lifting, and all the steps/tools needed to get the end result are all recorded and "replayable." Things that use to be laborious are now trivial. For example, a lot of DVDs put the audio commentary in a separate title (very annoying). With my language, I just specify the title as something like (1+2), and it knows to extract both titles with MakeMKV, use mkvmerge to combine all the tracks from the second (except the video and chapters) into the first, than add a batch entry to transcode the result with HandBrake (with my AutoIt tool, all those steps had to be written separately). I can also tell it to concatenate the titles instead of merging them.

It's nice because sometimes there is that one disc with some weird special features track that no tool but one can access (I like special features), be it CloneDVD, MakeMKV, DVDShrink, etc. The effort needed to access that track is preserved (because I will never remember!).


DrX

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Sun Sep 03, 2017 8:04 am

Thanks for the infos. I don't know if a method like this would work for me. I noticed that the tracks are often at various positions. So I never can tell, which track I would need in advance. Of course storing the job information might make it really easy to just change one setting eg. switch from h.264 to h.265.

I have a stack of media lying around which I have to care for. I think I will try out to go with the intermediate state of creating an mkv remux for those. Then I can make a better comparison of how big a remux and the reencode will be. I get some doubts the last days about my handbrake skills, since for many options I just go with the defaults or with some hints from the web. The idea of having the data not badly optimized is appealing.

Can you explain in more details how you store the extras now? I seldom extract this stuff, but really rarely it's gold worth. An extra audiotrack is of course easy but some cut outs etc. lead to knew movie files. For me it ended with having the same movie name several times in Kodi.

DrXenos
Bright Spark User
Posts: 209
Joined: Sat Mar 16, 2013 1:19 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by DrXenos » Sun Sep 03, 2017 12:12 pm

FordPrefect wrote:
Sun Sep 03, 2017 8:04 am
Thanks for the infos. I don't know if a method like this would work for me. I noticed that the tracks are often at various positions. So I never can tell, which track I would need in advance. Of course storing the job information might make it really easy to just change one setting eg. switch from h.264 to h.265.
The relative positions of tracks within a title are irrelevant (as for as I'm concerned). I go by the language and codex of a track to determine if I want it. When I process a script, the first time I encounter a source, I analyze it with HandBrake (or others) and parse the output. I cache this info for the next time I encounter the same source.
FordPrefect wrote:
Sun Sep 03, 2017 8:04 am
Can you explain in more details how you store the extras now? I seldom extract this stuff, but really rarely it's gold worth. An extra audiotrack is of course easy but some cut outs etc. lead to knew movie files. For me it ended with having the same movie name several times in Kodi.
Sorry, I am not sure what you are asking.

DrX

FordPrefect
Posts: 15
Joined: Mon Dec 19, 2016 9:44 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by FordPrefect » Sun Sep 03, 2017 12:18 pm

As far as I understood your last sentence from the post before you found a nice way to deal with extra content eg. a making of or whatsoever.

But maybe I got you wrong there.

DrXenos
Bright Spark User
Posts: 209
Joined: Sat Mar 16, 2013 1:19 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by DrXenos » Sun Sep 03, 2017 1:23 pm

The point of that sentence was I've seen a lot of weird ways special features are placed on DVDs (Blu-rays seem to be pretty straightforward). Sometimes it took a lot of effect to extract them, and I didn't want to forget how it was done. I've seen special features that only one particular tool seems to be able to access (and it's never the same one). Some examples:

1. Special features in menus: Sometimes DVDs (especially older ones), will have a trailer or featurette buried in the menus, and not in a title. HandBrake (HB) cannot access these (that I know of). The only tool I've found that can (without processing the vob files directly) is DVDShrink. So, in this case I'll script a call to DVDShrink to extract the video from the menu and then use HB to transcode it.

2. Multiple videos in one title. I don't know why DVD creators do this, but HB can't access any but the first. For these, I give my script a video number in addition to the title number. It knows to extract it using MakeMKV (because it can handle this) and then transcode with HB.

3. Audio-only tracks. Some DVDs have, as special features, audio-only tracks (e.g., radio broadcasts). HB can't handle these, but VLC and DAE can (I like the latter). So, I'll add a script line to extract these into mp3.

4. Hard to reach titles. I don't know what to call these, or why they occur, but I have encountered DVD titles that I can "see" and play in VLC, etc., but tools like HB just can't see them. Usually, MakeMKV can, but I've seen some that nothing could see but DVDShrink. For these, all extract with whatever tool can see them, and transcode with HB as usual.

I guess what I am getting at is, if you think you'll want to re-transcode your huge collection in the future, find a way to preserve all the time and effort it took to do it the first time (especially if you like special features--if you just like the main feature, it's not as bad). In my case, if I ever want to redo my collection in H.265, I just need to update my tool and rerun the scripts.

Hope that helps,
DrX

Woodstock
Veteran User
Posts: 3293
Joined: Tue Aug 27, 2013 6:39 am

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by Woodstock » Sun Sep 03, 2017 4:29 pm

Then there are curmudgeons like me who rip everything over 30 seconds, hand the main feature over to handbrake for encoding, watch the extras, and dispose of them afterward. It's rare that an extra really adds to the story (which is why I bought the movie or series), so why bother?

DrXenos
Bright Spark User
Posts: 209
Joined: Sat Mar 16, 2013 1:19 pm

Re: Upgrade of MKV collection from H.264 to H.265 possible?

Post by DrXenos » Sun Sep 03, 2017 4:38 pm

Woodstock wrote:
Sun Sep 03, 2017 4:29 pm
Then there are curmudgeons like me who rip everything over 30 seconds, hand the main feature over to handbrake for encoding, watch the extras, and dispose of them afterward. It's rare that an extra really adds to the story (which is why I bought the movie or series), so why bother?
So, I like to keep them and you don't. What difference does it make? I'm not interested in arguing over personal preference.

Post Reply