Hello,
I search some solution to support a lots of encoding process in a really short period of time, with as less administration as possible.
The CLI Handbrake will be a realy good solution. My experience with it show that's a nice solution with good quality with few bug (this f*cking sound drop sometimes make me crazy but ...).
So my question is can Handbrake support OpenMOSIX cluster (process migration between a cluster of computer) ?
Has I know OpenMOSIX, it works by moving processes from a heavily loaded machine to a more lightly loaded machine. Handbrake use only one process, but many thread, has I understand ?
Thanks for your help,
Handbrake (linux) with OpenMosix Cluster
I am doing this on a loose heterogeneous cluster of linux boxes and osx boxes, using perl and the CLI versions of handbrake/mediafork. It works rather well, and is surprisingly easy to do if you know a scripting language. I've also done it with XGrid as a grid controller.
Here's what I did:
1) Put all of your VIDEO_TS folders on a networked drive(s).
2) Put the cli binary in the same location on each machine.
3) write a script to find all of the titles and all of the VIDEO_TS folders on the drive(s)
4) write a script to control the job distribution (this could be as easy as a loop with ssh/rsh -e calls).
If you are using OpenMosix, I assume that you are familiar with some scripting language, so you shouldn't have any problems.
I have a script on this site somewhere that finds VIDEO_TS and titles. Since I process an entire hard drive at once on each machine (which takes days), I didn't see much need to get fancy with queuing, just "ssh/rsh -e mediaforker.pl /Volumes/Drive#" from the controller to all of the machines.
My script processing unit is a title, but if you wanted to process in smaller chunks and send them out to different machines, it could be easily modified to process by chapter (maybe even blocks or chunks?), but then you would have to write something to reassemble the chapters back into a movie.
Here is the post with the scripts:
http://handbrake.m0k.org/forum/viewtopic.php?t=1098
I think some of the developers may have recently written some useful scripts, as well, but I haven't used them yet.
Here's what I did:
1) Put all of your VIDEO_TS folders on a networked drive(s).
2) Put the cli binary in the same location on each machine.
3) write a script to find all of the titles and all of the VIDEO_TS folders on the drive(s)
4) write a script to control the job distribution (this could be as easy as a loop with ssh/rsh -e calls).
If you are using OpenMosix, I assume that you are familiar with some scripting language, so you shouldn't have any problems.
I have a script on this site somewhere that finds VIDEO_TS and titles. Since I process an entire hard drive at once on each machine (which takes days), I didn't see much need to get fancy with queuing, just "ssh/rsh -e mediaforker.pl /Volumes/Drive#" from the controller to all of the machines.
My script processing unit is a title, but if you wanted to process in smaller chunks and send them out to different machines, it could be easily modified to process by chapter (maybe even blocks or chunks?), but then you would have to write something to reassemble the chapters back into a movie.
Here is the post with the scripts:
http://handbrake.m0k.org/forum/viewtopic.php?t=1098
I think some of the developers may have recently written some useful scripts, as well, but I haven't used them yet.
My idea is to launch process only on the master machine and let OpenMOSIX migrate te process to other node.
Each node boot on a custom Dyne Bolic Live CD. I need to build this CD first with all the needed dependencies.
To deal with handbrake the idea is to use the CLI with -C 1 and launch one process per CPU (core) avaible on the cluster. The problem is to report the to global fps of the cluster and monitore all the process status from one console. Maybe some pipe with awk and grep can solve those issues. I need to be able to kill any process on the cluster if I suspect an audio drop, or any other problems.
I use OpenMOSIX to render some PureData calculation and blender rendering with success.
The big advantage of OpenMOSIX is the flexibility on the number of node. When I go home, I can add my portable computer to the cluster, by simply boot it on a Live CD (Dyne Bolic) and my clusteris extented automaticaly. So i can use any computer( girlfriend's computer, friend's computer
)
And OpenMOSIX balance the load really well on the different node.
Each node boot on a custom Dyne Bolic Live CD. I need to build this CD first with all the needed dependencies.
To deal with handbrake the idea is to use the CLI with -C 1 and launch one process per CPU (core) avaible on the cluster. The problem is to report the to global fps of the cluster and monitore all the process status from one console. Maybe some pipe with awk and grep can solve those issues. I need to be able to kill any process on the cluster if I suspect an audio drop, or any other problems.
I use OpenMOSIX to render some PureData calculation and blender rendering with success.
The big advantage of OpenMOSIX is the flexibility on the number of node. When I go home, I can add my portable computer to the cluster, by simply boot it on a Live CD (Dyne Bolic) and my clusteris extented automaticaly. So i can use any computer( girlfriend's computer, friend's computer
)And OpenMOSIX balance the load really well on the different node.