Cluster / multiple machines cpu encoding

Archive of historical feature requests.
Please use the GitHub link above to report issues.
Forum rules
*******************************
Please be aware we are now using GitHub for issue tracking and feature requests.
- This section of the forum is now closed to new topics.

*******************************
Post Reply
Skramse
New User
Posts: 1
Joined: Sun Jan 01, 2017 3:45 am

Cluster / multiple machines cpu encoding

Post by Skramse » Sun Jan 01, 2017 3:54 am

Cluster cpu encoding.
Don't know if it's possible but it would be awesome to use cpu from multiple machines in the same network to make encoding faster. This feature is commonly found in 3D rendering softwares


I have an army of laptop available so instead of start different jobs on multiple machines it would be nice to just que them all in the same place and use the power from the other machines as well :

Latest x64 portable version

Windows 10 mostly. Would be awesome if different operating systems could be used in the same cluster

musicvid
Veteran User
Posts: 3272
Joined: Sat Jun 27, 2009 1:19 am

Re: Cluster / multiple machines cpu encoding

Post by musicvid » Sun Jan 01, 2017 1:25 pm

You would be disappointed. Distributed encoding involves a lot of maintenance for very little net speed gain. It is also incredibly complicated for long-gop interframe encoding, which your 3d programs are not. Multiple laptops with dissimilar CPUs would make your encoding go slower, not faster than a single machine.

Woodstock
Veteran User
Posts: 3183
Joined: Tue Aug 27, 2013 6:39 am

Re: Cluster / multiple machines cpu encoding

Post by Woodstock » Sun Jan 01, 2017 3:59 pm

Not to mention that laptop cooling systems are not up to the task of running at 100% CPU for long periods of time.

musicvid
Veteran User
Posts: 3272
Joined: Sat Jun 27, 2009 1:19 am

Re: Cluster / multiple machines cpu encoding

Post by musicvid » Mon Jan 02, 2017 9:06 am

Oh no, not in this case, because most of the laptops would jus sit idling most of the time waiting for the slowest cpus to catch up! Tanstaafl

Woodstock
Veteran User
Posts: 3183
Joined: Tue Aug 27, 2013 6:39 am

Re: Cluster / multiple machines cpu encoding

Post by Woodstock » Mon Jan 02, 2017 3:10 pm

Catch up or burn up... :)

WhosAsking
Posts: 45
Joined: Fri Jul 03, 2009 1:40 pm

Re: Cluster / multiple machines cpu encoding

Post by WhosAsking » Wed Mar 08, 2017 4:56 pm

The main problem is in task delegation, especially in a one-pass or first-pass encode where the encoder doesn't know how it plans to structure the GOPs. There was a similar discussion in the x264 board when they were going into multi-core encoding. Basically, trying to delegate by GOPs involves a whole lot of guesswork, plenty of which can be mistaken because it can't know ahead of time where the keyframes are located (a scene change can cue a keyframe early, and it can't rely on chapter points that may not be there). Plus unlike 3D farm rendering, video encoding is a dependent task. You use the previous data to help with the next frame, whereas rendering techniques like ray tracing don't need to know what happened with the previous ray.

musicvid
Veteran User
Posts: 3272
Joined: Sat Jun 27, 2009 1:19 am

Re: Cluster / multiple machines cpu encoding

Post by musicvid » Wed Mar 08, 2017 5:12 pm

That is essentially correct. B-frames need information both upstream and downstream.

nhyone
Bright Spark User
Posts: 221
Joined: Fri Jul 24, 2015 4:13 am

Re: Cluster / multiple machines cpu encoding

Post by nhyone » Thu Mar 09, 2017 1:57 am

The original poster is asking about using one computer to distribute jobs to other computers, not breaking up one video into chunks.

Something like this? :D

Although once you have this, it is not a stretch to think about distributed encoding... especially for HEVC...

mduell
Veteran User
Posts: 6630
Joined: Sat Apr 21, 2007 8:54 pm

Re: Cluster / multiple machines cpu encoding

Post by mduell » Thu Mar 09, 2017 2:39 am

Your screenshot of a dashboard solves none of the distributed encoding problem, HEVC or otherwise.

nhyone
Bright Spark User
Posts: 221
Joined: Fri Jul 24, 2015 4:13 am

Re: Cluster / multiple machines cpu encoding

Post by nhyone » Thu Mar 09, 2017 3:02 am

mduell wrote:
Thu Mar 09, 2017 2:39 am
Your screenshot of a dashboard solves none of the distributed encoding problem, HEVC or otherwise.
That's right, it only distributes whole jobs to a cluster of machines from a central machine.

But it doesn't help when you have fewer videos than machines. Suppose you only have one video and 30 encoders. There is no speedup at all.

IMO, it should be possible to break the video into, say 5-min segments (this can't be too fine), encode them in parallel, then merge the parts together.

Post Reply