Splitting files for faster/higher encodes?

General questions or discussion about HandBrake, Video and/or audio transcoding, trends etc.
Post Reply
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Splitting files for faster/higher encodes?

Post by YaBoyShredderson »

My understanding is that x264 can fully utilize up to 8 cores/16 threads, and x265 is maxed out at lower core/thread counts. I also know that higher thread counts reduces quality at a given bitrate, though its negligible until you get lower than about 40 vertical pixels per thread.

If i have for example, a 12 core/24 thread cpu, would it be more beneficial for me to split the file into 12 roughly equal parts and run 12 instances of handbrake with threads=2 in the advanced options box, and then stitch them back later? Or will this cause issues? I thought about doing this for 4k content with x265 as it takes a god awful amount of time to do, upwards of 20 hours, and this would allow me to encode a small portion for maybe 2 hours each, allowing my pc to be used for other things/people in the meantime, or just shut down entirely. I would not be encoding using only 2 threads in that scenario though.
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Splitting files for faster/higher encodes?

Post by rollin_eng »

You are going to give yourself more issues if you start chopping up your videos then try to put them back together.

Just trust the encoders to do what’s best.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Splitting files for faster/higher encodes?

Post by YaBoyShredderson »

rollin_eng wrote: Sun Aug 23, 2020 12:12 pm You are going to give yourself more issues if you start chopping up your videos then try to put them back together.

Just trust the encoders to do what’s best.
Its far too slow to do it that way for 4k, with 1080p i have just been waiting it out. But 4k it encodes at around about or even less than 1 frame per second. As long as i leave the settings exactly the same, then i would thing i wouldnt have issues. What isses do you think i would have?
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Splitting files for faster/higher encodes?

Post by rollin_eng »

Your biggest issues will be where to split your file and then maybe syncing issues when you put it back together.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Splitting files for faster/higher encodes?

Post by YaBoyShredderson »

rollin_eng wrote: Sun Aug 23, 2020 12:31 pm Your biggest issues will be where to split your file and then maybe syncing issues when you put it back together.
Why would it matter where i split it? Mkvtoolnix has no choice but to split on an I frame, so there should be no issues there, and csnt u just cut into say, 5 roughly even clips?
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Splitting files for faster/higher encodes?

Post by rollin_eng »

I dont know Mkvtoolnix so I cant speak for its accuracy.
Deleted User 13735

Re: Splitting files for faster/higher encodes?

Post by Deleted User 13735 »

Why don't you go ahead and try? Then post your data in a quantified manner. Quantified means real numbers on a level playing field with a control set. I predict you will learn some things about distributed rendering and overhead that we explored fully two decades ago.
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Splitting files for faster/higher encodes?

Post by rollin_eng »

If I remember right ripbot264 has a feature to distribute encodes to multiple PC's.

Not exactly what you want but might be worth looking at.
Deleted User 11865

Re: Splitting files for faster/higher encodes?

Post by Deleted User 11865 »

Depending on the audio format, video and audio frame boundaries may not align exactly, so you'll want to be extra careful when splitting/joining files to avoid compounding A/V sync offsets.

Generally speaking, it's just easier for most end users to encode multiple different movies/episodes at once, rather than splitting a single source and encoding it in chunks. Large VOD providers (Netflix, Amazon, YouTube, Vimeo etc.) sometimes do the latter, but they all employ multiple full-time specialist developers who write custom software specifically for that purpose.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Splitting files for faster/higher encodes?

Post by YaBoyShredderson »

Rodeo wrote: Sun Aug 23, 2020 6:32 pm Depending on the audio format, video and audio frame boundaries may not align exactly, so you'll want to be extra careful when splitting/joining files to avoid compounding A/V sync offsets.

Generally speaking, it's just easier for most end users to encode multiple different movies/episodes at once, rather than splitting a single source and encoding it in chunks. Large VOD providers (Netflix, Amazon, YouTube, Vimeo etc.) sometimes do the latter, but they all employ multiple full-time specialist developers who write custom software specifically for that purpose.
I did some testing with 1 minute clips, encoding 10 seconds and then stitching them. 2 out of 3 had audio issues at the 30sec mark, but no other stitches had them. So 15 joins in total only 2 had audio glitches, it still remained synced but one them sounded like there was a bit of static and the other cut out for a split second. Would encoding the audio seperately help prevent this? The video didnt seem to have any issues, so doing them seperately could help.

Doing multiple encodes at the same time isnt really an option, it was more of a bonus thing letting me better utilize my cpu for other stuff that i would be willing to wait around for anyway. With 4k stuff, it can take in excess of a day to encode, and i simply cannot wait that long, as my brother also uses my pc. But i can finish a 2hour encode, and then do another one later which can all be stitched together, which the original region for asking this question.
Deleted User 11865

Re: Splitting files for faster/higher encodes?

Post by Deleted User 11865 »

Convince your parents to send your brother to boarding school :twisted:
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Splitting files for faster/higher encodes?

Post by rollin_eng »

You could also just move the encoder preset to a faster setting.
Post Reply