Distributed/Parallel x264 encoding?

General questions or discussion about HandBrake, Video and/or audio transcoding, trends etc.
Post Reply
gorkypah
Posts: 3
Joined: Fri Jun 04, 2010 6:17 am

Distributed/Parallel x264 encoding?

Post by gorkypah »

I just was wondering if Handbrake or any other Windows video encoders had support for distributed encoding over a network? I know Linux has DVD::RIP which has cluster support, but it takes so much configuration to get it to work (working SSH and NFS for example). I just think it would be useful if Handbrake or other Windows video encoders had support for encoding over a network? It would make the encoding process so much faster, especially for encoding a DVD collection.
davidfstr
Regular User
Posts: 149
Joined: Sun Apr 12, 2009 7:41 pm

Re: Distributed/Parallel x264 encoding?

Post by davidfstr »

HandBrake does not support distributed network encoding. Certainly a difficult feature to implement.
musicvid
Veteran User
Posts: 3963
Joined: Sat Jun 27, 2009 1:19 am

Re: Distributed/Parallel x264 encoding?

Post by musicvid »

It would make the encoding process so much faster,
The advantages of distributed rendering, even with identical CPUs over a gigabit network, are so modest as to be not worth the effort. The overhead is the equalizing factor. Take my word.
magao
Posts: 14
Joined: Thu Jul 08, 2010 11:10 pm

Re: Distributed/Parallel x264 encoding?

Post by magao »

What would be useful would be a distributed job queue. Basically, setting up slave instances of Handbrake on other machines, that have jobs allocated to them as they become available.
louwrentius
Posts: 13
Joined: Thu Aug 05, 2010 11:57 pm

Re: Distributed/Parallel x264 encoding?

Post by louwrentius »

It requires a little bit of setup, but you could use multiple computers using PPSS:

http://forum.handbrake.fr/viewtopic.php?f=6&t=17504
Firelight
Posts: 24
Joined: Fri Aug 13, 2010 9:06 pm

Re: Distributed/Parallel x264 encoding?

Post by Firelight »

I have an xgrid cluster of mac minis uniformly encoding to 720p with all computers going to and from an NFS mount. This is with a distributed job queue, not single file parallelization. Works quite well. You need to get comfortable with the CLI tho and xgrid is a little quirky.
dynaflash
Veteran User
Posts: 3820
Joined: Thu Nov 02, 2006 8:19 pm

Re: Distributed/Parallel x264 encoding?

Post by dynaflash »

fwiw, current nightlies support multiple instances off of the same queue on *the same computer* ... next would be to get paths and pids via bonjour or the like to make it work across a lan. No guarantees. Just an fyi.
louwrentius
Posts: 13
Joined: Thu Aug 05, 2010 11:57 pm

Re: Distributed/Parallel x264 encoding?

Post by louwrentius »

Just installing handbrake (cli?) on some nodes and point them all to a stack of files that get processed, while preventing 'collisions' would be great. One could be the master and the other ones the slaves. Would be way cool.
gt46l
Posts: 3
Joined: Tue Jan 18, 2011 3:01 pm

Re: Distributed/Parallel x264 encoding?

Post by gt46l »

I know this thread hasn't been touched in a few months, but I figured I would use it instead of creating a new one. There is a cluster encoder out there. Its a german site but they post in English: http://codergrid.de Distributed batch encoding is nice if you have 20 files to encode, but cluster encoding is the way to go if you have one 40gb file to encode. Their encoding interface and presets system sux, but is easy enough to use, and it works on Mac, Linux, Windows. Handbrake has everything they don't have. The ultimate and obvious solution would be to combine efforts/merge projects and focus their development since they are trying to rollout too many features at once. I'm betting most people using an encoder these days have multiple computers sitting around at home. I personally have 7 computers for a total of 14 cores with a gigabit network in my house. Worse case scenario If I only gained 10% speed increase for each additional computer it would take less than half the time to encode a 40gb file. I also work in IT for a major healthcare provider in the US. We are literally throwing out all of our new Dell servers because they don't have a stupid TPM chip in them needed for security due to HIPAA regs..... I get all the free dual core Xeon servers I want. I'm certainly motivated to try a cluster encoder!
Post Reply