Cluster / multiple machines cpu encoding

[Skramse]

Cluster cpu encoding.
Don't know if it's possible but it would be awesome to use cpu from multiple machines in the same network to make encoding faster. This feature is commonly found in 3D rendering softwares

I have an army of laptop available so instead of start different jobs on multiple machines it would be nice to just que them all in the same place and use the power from the other machines as well :

Latest x64 portable version

Windows 10 mostly. Would be awesome if different operating systems could be used in the same cluster

[Deleted User 13735]

You would be disappointed. Distributed encoding involves a lot of maintenance for very little net speed gain. It is also incredibly complicated for long-gop interframe encoding, which your 3d programs are not. Multiple laptops with dissimilar CPUs would make your encoding go slower, not faster than a single machine.

[Woodstock]

Not to mention that laptop cooling systems are not up to the task of running at 100% CPU for long periods of time.

[Deleted User 13735]

Oh no, not in this case, because most of the laptops would jus sit idling most of the time waiting for the slowest cpus to catch up! Tanstaafl

[Woodstock]

Catch up or burn up...

[WhosAsking]

The main problem is in task delegation, especially in a one-pass or first-pass encode where the encoder doesn't know how it plans to structure the GOPs. There was a similar discussion in the x264 board when they were going into multi-core encoding. Basically, trying to delegate by GOPs involves a whole lot of guesswork, plenty of which can be mistaken because it can't know ahead of time where the keyframes are located (a scene change can cue a keyframe early, and it can't rely on chapter points that may not be there). Plus unlike 3D farm rendering, video encoding is a dependent task. You use the previous data to help with the next frame, whereas rendering techniques like ray tracing don't need to know what happened with the previous ray.

[Deleted User 13735]

That is essentially correct. B-frames need information both upstream and downstream.

[nhyone]

The original poster is asking about using one computer to distribute jobs to other computers, not breaking up one video into chunks.

Something like this?

Although once you have this, it is not a stretch to think about distributed encoding... especially for HEVC...

[mduell]

Your screenshot of a dashboard solves none of the distributed encoding problem, HEVC or otherwise.

[nhyone]

Your screenshot of a dashboard solves none of the distributed encoding problem, HEVC or otherwise.

That's right, it only distributes whole jobs to a cluster of machines from a central machine.

But it doesn't help when you have fewer videos than machines. Suppose you only have one video and 30 encoders. There is no speedup at all.

IMO, it should be possible to break the video into, say 5-min segments (this can't be too fine), encode them in parallel, then merge the parts together.