Optimal PC Hardware for Handbrake conversions

[dquill]

Hi,

This is my first post here, but hope to find good ideas on the question. Basically, I've been using Handbrake (in batch mode) to convert 100s of videos a day. Recently we started receiving more video which is going to force me to build a larger box to maintain this video.

The question is fairly simple, what does Handbrake like the most in hardware? Memory, Processing Speed, hard drive speed I'm assuming, but I'd like to consider it in a bit more detail. If I get 20 Gigs of memory vs. 10, would I see any increase? Should I look at solid state drives for the video processing?

The job we are doing is converting up to a 15000 small videos a day. These video's range from 2 - 3 Megs each and are in various formats (AVI, VC1 in an ASF wrapper, MPG being the most frequent). I basically create a batch file each evening with all the files that need processing. We've been doing this for a while, and it works great, but I'm hoping to speed the process up as much as I can. Currently it runs at near real time, if a clip is a minute, it takes a minute ten to convert (this is apprx.).

Thanks for any help you may provide.

[s55]

Memory
SD: 1 to 200MB
HD: 500 to 2000MB

Dependant on settings.
In other words, a system with 8GB is probably fine as long as it's not being used by other software.

Hard Drive
The only time you'll see a benefit to a faster drive, is if you are running more than a couple of encodes at the same time. It's only when you have multiple reads at the same time does drive end up having to seek an awful lot more degrading performance.

CPU

Stick with the fastest Intel CPU with as many cores as your budget will allow.
Depending on your encode settings, you may find that running multiple instances of HandBrake will better utilise the CPU.

[dquill]

s55,good points. I'll have to evaluate if we'll run multiple instances. So by the sound of it, CPU is likely where we will find the greatest impact. I hadn't thought about doing multiple threads. My budget for this could be fairly decent, maybe 7K or more if I need it. Generally I'd just buy a top end workstation type box to do the job. Seems like the Intel Xeon E5-2687W @ 3.10GHz is performing pretty well, but it's fairly expensive.

Historically, we have encoded a single file at a time. Our main concern is the volume is up by 50x or more so coming in with the right hardware should have a significant impact. This box I'm putting together will only do encoding, it won't even have a keyboard of monitor, rather just Remote Desktop.

The source of the video is coming off a NAS, and it will eventually get back to the NAS.

[rollin_eng]

Your network/NAS may also be a bottleneck.

I would also consider getting multiple machines, maybe 5 x 1.5k this way you have redundancy too.

[mduell]

NAS for storage shouldn't be an issue unless it's complete [Censored].

With a $7k budget you will probably be better off with multiple single socket machines unless there's significant management overhead (non-automated).

[GregiBoy]

I agree with the multiple machines and there are plenty of scripts here on the forum that check for new files in a directory and add them to the queue.

All you would have to do then is set up an input directory for each machine and output to a common directory and drop the files into the relevant directory.

Only thing that you would have to be very circumspect about is your file naming cnvention.

[dquill]

The main restriction I have is more about physical space. I'm going to evaluate if I can actually fit that many boxes.

You wouldn't recommend using a VM for this? We have virtualized our servers, and could potential virtualize a series of processing boxes, but in general the virtual boxes don't seem quite as efficient as real hardware. I could very likely get as many virtual boxes as I need for the task.

Scripting the data is not a problem. We've managed that for years, so getting that part done is easy. We basically injest the file directories into a database which checks what's been done and what hasn't. I then create a batch job that basically converts with our settings. The one good thing is we rename any files that are inappropriately named. It's a fairly involved process, but it's entirely mastered now.

[GregiBoy]

As you have noted VM's are just not as efficient as stand alones. I would stick with standalones.

To save space, you could use small footprint boxes, maybe racked up with one keyboard and monitor all connected by a KVM switch for emergency use and actually control them by a remote desktop.

I did have something similar a few years ago doing a different set of tasks but it worked for me.

[s55]

Mini-ITX boxes might be the answer if space is a problem.

[mduell]

If you have easy access to VMs that's not a bad idea. I wouldn't set them up for this purpose, but when they already exist they're easy. The performance hit isn't that much.

[s55]

The overhead with VMWare isn't too bad. VirtualBox is pretty bad though.

[mduell]

Thankfully no one is running many servers on VirtualBox. It's not even stable on the desktop.

[dquill]

I believe we are using VM Ware as our virtual software environment. I know the VM servers I currently have running don't convert the video nearly as quickly as our workstations do. The speed difference is maybe half the speed of the workstation. As servers, we've had no issues with the VMs, they run most of our application infrastructure but video conversions is a different animal. It's a "simple" way to answer the many machine question, but I'm not sure it's the right answer. We seem to have differing opinions on how well the VM will perform but it certainly solves the space issues.

The three routes I have now are:

1. Big Workstation. Most don't seem to think this is the best route to go
2. Multiple smaller machines. I'm checking into size, but this is really challenging from a space manager perspective, but likely the best technical approach.
3. Multiple smaller VMs. Not efficient, but answers the multiple processing box question. It would mean half or less our current speed of a workstation.

I'm struggling with the decision now. Originally I thought a bigger faster box would suffice, but this discussion has shown that thought likely incorrect.

[mduell]

Do the VMs have the same number and speed of CPUs as the workstations?

[Tree Dude]

I have played around with encoding using a VM on VMware ESXi 5.0. It was nowhere near as fast as a comparable physical box. The way ESXi handles the CPU scheduling just does not play well with encoding. Stick to a physical box with no more than 8 threads (unless you want to run multiple instances on the same box).

[unixguy55]

My ranking of importance for equipment is follows:

Clock speed
Cores
RAM

My primary system is a dual quad-core Xeon E5450 @3ghz w/ 16GB RAM. My system before that was the same architecture CPU, but single quad-core processor at 3ghz w/8GB RAM. I am seeing roughly 2-3x improvement in encoding times with this system. Another box I have tested is a socket 1366 Dell Precision workstation with 2 processors @ 1.6ghz and quad cores w / 12GB RAM. The Precision is less than half as fast at encoding as my older Core2-based Xeon system, with the same number of physical cores.

Go Intel for the processor and get as high of clock speed with the largest number of cores you can afford. I would say at least 8GB RAM for each instance of the encoder you want to run, if running multiples, depending on what else you do with the system.