Want to reduce transcode time...

[rpurinton]

Hi all! I'm building a 'massive' transcoder box and I'm looking for help optimizing and improving transcode performance. This is for the media sharing website RushTera.com. Users upload large raw videos often times greater than 10GB in size. When sharing a video we run it through transcoding to prepare the file for web streaming via Wowza media server. We create 4 transcodes for each shared video, with heights of 1080, 720, 480, and 360p.

Currently we are telling users that transcodes generally take as long as watching the video in real time, so if the user uploaded a 2 hour movie, our goal is to transcode it into 4 streamable mp4s in 2 hours or less.

To accomplish this, we've ordered up a 'massive' server from Softlayer
Quad Intel Xeon E7-4850v2 (12 core / 24 thread 2.3Ghz). When I cat /proc/cpuinfo it shows 96 CPUs. The system also has 512GB DDR3 2133, and 4 x 800GB SSD's in RAID10.

We're running CentOS 6.7 64-bit on the system: 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I just compiled the latest version of HandBrake from github, 0.10.3. When compiling, I did ./configure --launch --disable-gtk

Everything compiled fine. But I did notice when running ./configure it said CPU count was 32...

Here are the commandlines we are currently using to create our proxies: (please note, i didn't come up with these, so I dont know if they are the best options to use... again our goal is to create Web Optimized mp4s for streaming with Wowza...

Code: Select all

/usr/local/bin/HandBrakeCLI -i '/root/4k.mp4' -o ~/4k-360p.mp4 --encoder x264 -O --decomb="fast" --loose-anamorphic --modulus 2 --vfr --x264-preset=medium --x264-profile=main -w 640 -q 22

/usr/local/bin/HandBrakeCLI -i '/root/4k.mp4' -o ~/4k-480p.mp4 --encoder x264 -O --decomb="fast" --loose-anamorphic --modulus 2 --vfr --x264-preset=medium --x264-profile=main -w 853 -q 22

/usr/local/bin/HandBrakeCLI -i '/root/4k.mp4' -o ~/4k-720p.mp4 --encoder x264 -O --decomb="fast" --loose-anamorphic --modulus 2 --vfr --x264-preset=medium --x264-profile=main -w 1280 -q 22

/usr/local/bin/HandBrakeCLI -i '/root/4k.mp4' -o ~/4k-1080p.mp4 --encoder x264 -O --decomb="fast" --loose-anamorphic --modulus 2 --vfr --x264-preset=medium --x264-profile=main -w 1920 -q 22

We execute all 4 of these at the same time, so there are 4 simultaneous HandBrakeCLI processes executing at the same time...

Here is a snapshot of atop during the transcoding:



As you can see, we are not redlining the CPU as I had expected... doing this on other servers we tested always sent the CPU to 100%. On this machine it looks like it's about 35% CPU usage...

Speed is our key factor we're going for here... Users spend a long time waiting for their upload to complete, then they submit them for transcoding and have to wait "in the dark" for the transcodes to complete. We don't currently have a way to display the transcode progress to them, so they just have to wait for completion. We are hoping to reduce our customer's wait times as much as possible.

Here's a few Benchmarks
1080p source video 10 minutes long 189MB transcoded all 4 proxies in 3 minutes and 10 seconds... This was great!
2160p source video 68 minutes long 7.8GB transcoded all 4 proxies in 74 minutes... this is great compared to all our previous transcodes, but still not under our target, and based on the results of the 1080p test, i was hoping this one would have completed in about 25 minutes or so, but that wasn't the case...

Some other thoughts... during these bench marks the files were read and written to the SSD drives which benchmark at about 1.5GB/sec so I dont think disk IO speed is a factor here...

Activity Log: http://pastebin.com/CsDX6an5

Only one question arises from that log: It says CPU count is 64... Should be 96.

OK all the information is out on the table, does anyone have any suggestion on how we could reduce our transcode times either by tweaking our ./configure commandline or our ./HandBrakeCLI command line?

Thanks in advance!

Russ Purinton
RushTera

[s55]

A single instance will typically scale well to around 6 cores. After that it can be a bit erratic.

So you should be able to spool up more instances on that machine assuming memory bandwidth is not an issue.

You can try using a faster x264 preset if you want more speed. If you don't need Decomb then removing it will also give a small speed up

[rollin_eng]

While your machine is a beast there are Xeons that have higher clock speeds (usually at the cost of less cores), so if you can get a different Xeon to test you may see a speed increase.

[nhyone]

See if the veryfast preset is acceptable for you. This is 3x the speed of medium, and often gives a small size that only veryslow can consistently beat. (veryfast uses trellis 0, and medium trellis 1, so it will look a little different.)

Based on your machine spec, it should have 4 CPUs, each with 12 cores?

If so, I would suggest using 6 cores per encoding task, so you can encode 8 videos at a time. You can use processor affinity to make the tasks run on designated cores, to avoid them from jumping around. (But be sure you set the affinity correctly! It is very inefficient to use cores on two different CPUs -- only 75% as fast in my experience.)

There are many combinations you can explore. For example, you can use 4 cores to encode 360p and 480p sequentially, 8 cores to encode 720p and 12 cores to encode 1080p. This uses just 2 CPUs, so you can repeat for the other 2 CPUs.

(I try not to use cores across two CPUs, but that's just me.)

I use mpstat to check that the cores are used as expected.

You can get the HandBrake progress if you redirect the standard output to a file and then tail that file to get the latest progress percentage.

[tlindgren]

Hi all! I'm building a 'massive' transcoder box and I'm looking for help optimizing and improving transcode performance. This is for the media sharing website RushTera.com. Users upload large raw videos often times greater than 10GB in size. When sharing a video we run it through transcoding to prepare the file for web streaming via Wowza media server. We create 4 transcodes for each shared video, with heights of 1080, 720, 480, and 360p.

Quad Intel Xeon E7-4850v2 (12 core / 24 thread 2.3Ghz). When I cat /proc/cpuinfo it shows 96 CPUs.

As others have indicated, a massive box will require several simultaneous encodings before it can be utilized. Also, you probably want to consider a more modern OS than CentOS 6, like CentOS 7. I *love* my C6 boxes but a lot of the included software really is too old (2006-era) for fast moving projects like video encoders.

In addition, 4-socket servers tend to be massively overpriced and often lags when it comes to new cores. In fact, I'd go as far as to claim that it's rare to see one give significantly better performance than a much cheaper top-end two-socket server, the quad socket boxes DO have a few special niches where they make sense (like massive max memory capacity and RAS features) but I really doubt video encoding is one of it's niches.

Based on the Softlayer's public pricing and without putting massive amounts of time into exploring all their options I'd guess that these would likely be relatively sane starting choices:

• Most transcoding performance per dollar: Single socket E3-1270v3 (4 core+HT, 3.5GHz)
• Fastest possible transcode: either the E3 above or a single socket E5-2690 (8 core+HT, 2.9GHz), which one is best depends on x264 profile and filtering options.
• "Good" transcode per dollar and much fewer machines to manage: Dual socket E5-2690v3 (dual 12-core+HT, 2.6GHz).

There are a number of other Xeons which would likely improve on these but isn't on Softlayer's public lists, but that should suffice to run some tests.

[Deleted User 11865]

Regarding 64 logical cores vs. expected 96, that does seem weird. Are all 96 cores recognized by the OS and other applications?

[rpurinton]

WOW! thanks everyone for all this feedback... It might take me a moment to parse thru it all...

Some general notes before I reply to specific things...

1) We have a technology "incubator" grant from SoftLayer where we get up to $10,000/mo of free services... this is why we got this massive server, because it's free for a year. Normally they would charge almost $5000 per month for it! Our hope is within the year we change to something else where we use on demand 32 core cloud instances for the transcoding. In the meantime this is our solution.

2) There are cases where we expect to transcode multiple files at once... ie: if a user submits a gallery of videos, we may have to Transcode 2, 3, 4, 5 or more videos simultaneously, which means 8, 12, 16, or 20 concurrent HandBrake processes (4 proxies per video). We've had cases in the past where this has crashed other servers (out of memory), we hope this one can handle many many many HandBrake processes.

3) I'm not sure I understand what effect these have:
-O --decomb="fast" --loose-anamorphic --modulus 2 --vfr

[rpurinton]

If you don't need Decomb then removing it will also give a small speed up

Unfortunately, my Video lingo isn't quite up to par (I'm just a Linux Guy thrown into the grinder!) I'm not sure what Decomb does and if we should use it or not for our proxies. Basically as I mentioned before, clients are uploading Hi-Res source video and we're transcoding it down for web streaming across various devices.

We definitely want to avoid our clients saying our streaming videos "looks bad"... I might look at a video with Decomb on and one with Decomb off and not be able to tell the difference, but we work with movie studios video professionals, I'm wondering if they'd be able to tell?

Honestly the same goes for the other CLI options we're using... I have no idea what their effect/impact is... like --loose-anamorphic, --vfr?

See if the veryfast preset is acceptable for you. This is 3x the speed of medium, and often gives a small size that only veryslow can consistently beat. (veryfast uses trellis 0, and medium trellis 1, so it will look a little different.)

I'll give that a try and see if the video pro's complain at all...

Based on your machine spec, it should have 4 CPUs, each with 12 cores?

Physical cores 48, Logical Cores 96 (Hyperthreading)

If so, I would suggest using 6 cores per encoding task, so you can encode 8 videos at a time. You can use processor affinity to make the tasks run on designated cores, to avoid them from jumping around. (But be sure you set the affinity correctly! It is very inefficient to use cores on two different CPUs -- only 75% as fast in my experience.)

There are many combinations you can explore. For example, you can use 4 cores to encode 360p and 480p sequentially, 8 cores to encode 720p and 12 cores to encode 1080p. This uses just 2 CPUs, so you can repeat for the other 2 CPUs.

(I try not to use cores across two CPUs, but that's just me.)

I use mpstat to check that the cores are used as expected.

Ok, I'll definitely experiment with this a bit.

You can get the HandBrake progress if you redirect the standard output to a file and then tail that file to get the latest progress percentage.

Good Idea!

While your machine is a beast there are Xeons that have higher clock speeds (usually at the cost of less cores), so if you can get a different Xeon to test you may see a speed increase.

We have tested on a wide variety of servers. I did find that a 4 core (8 thread) Xeon E3 3.7 Ghz did pretty well for running 4 simultaneous HandBrake jobs, however when we loaded it up with 8, 12, 20, 24 etc it took a very long time to complete.

As others have indicated, a massive box will require several simultaneous encodings before it can be utilized. Also, you probably want to consider a more modern OS than CentOS 6, like CentOS 7. I *love* my C6 boxes but a lot of the included software really is too old (2006-era) for fast moving projects like video encoders.

We run 4 simultaneous encodings per submitted video, and sometimes users submit multiple videos simultaneuosly, so we may be running 4,8,12,16,20... HandBrakeCLI's at once.

CentOS 6 was chosen simply because all of our other hundreds of servers are on it, and I feel 99% comfortable with it, whereas CentOS 7, I only feel about 30% confident supporting it... they changed so many things and I just haven't figured out all the differences... If you know a good guide for converting C6 admins into C7 admins I'd happily try it out, if you think this will boost our encoder performance.

In addition, 4-socket servers tend to be massively overpriced and often lags when it comes to new cores. In fact, I'd go as far as to claim that it's rare to see one give significantly better performance than a much cheaper top-end two-socket server, the quad socket boxes DO have a few special niches where they make sense (like massive max memory capacity and RAS features) but I really doubt video encoding is one of it's niches.

Based on the Softlayer's public pricing and without putting massive amounts of time into exploring all their options I'd guess that these would likely be relatively sane starting choices:

• Most transcoding performance per dollar: Single socket E3-1270v3 (4 core+HT, 3.5GHz)
• Fastest possible transcode: either the E3 above or a single socket E5-2690 (8 core+HT, 2.9GHz), which one is best depends on x264 profile and filtering options.
• "Good" transcode per dollar and much fewer machines to manage: Dual socket E5-2690v3 (dual 12-core+HT, 2.6GHz).

There are a number of other Xeons which would likely improve on these but isn't on Softlayer's public lists, but that should suffice to run some tests.

I agree... this server's price is ridiculous, almost $4500 per month, but it's free for us for a year thanks to SoftLayer's tech incubator grant... So with everything being free, we just got the biggest box they have!

[rollin_eng]

Speed and cores:

http://ark.intel.com/products/75258/Int ... e-2_80-GHz

You say that you are not a video expert but your business is about encoding/storing videos, you must have one somewhere I suggest you get him/her in the loop.

Cropping will also speed up encoding if needed.

[rpurinton]

Speed and cores:

http://ark.intel.com/products/75258/Int ... e-2_80-GHz

You say that you are not a video expert but your business is about encoding/storing videos, you must have one somewhere I suggest you get him/her in the loop.

Cropping will also speed up encoding if needed.

Thanks, I only know enough of video encoding to be dangerous! And yes, we have several video 'experts' available... I think I'll need to create the proxies on veryfast and then see if they have any complaints! I wish we could get that Proc too, but this is the fastest one offered.

Something else I've just discovered: It seems the encodings go much faster when the source video is first copied to a Ramdisk, and the output also written to the RamDisk... We have 512GB in this machine, so the default /dev/shm Ramdisk is already 256GB which should be plenty to store the source and the output!

[Deleted User 11865]

Something else I've just discovered: It seems the encodings go much faster when the source video is first copied to a Ramdisk, and the output also written to the RamDisk... We have 512GB in this machine, so the default /dev/shm Ramdisk is already 256GB which should be plenty to store the source and the output!

That's unexpected, TBH. How much faster exactly? I wouldn't mind seeing the encode logs for both (same file, same settings, ram disk vs. SSD).

FWIW, due to the x264 lookahead and some of the other things we buffer, in some cases HandBrake might use up to ~10 GB of RAM per instance when encoding 4K video (~2 GB for 1080p).

[rollin_eng]

Thanks, I only know enough of video encoding to be dangerous! And yes, we have several video 'experts' available... I think I'll need to create the proxies on veryfast and then see if they have any complaints! I wish we could get that Proc too, but this is the fastest one offered.

While I don't know your business model, using veryfast will speed things up at a cost of filesize and/or quality. Which if you are streaming cant be good.

I guess an option might be to allow your users to tweak some of the settings. If they are pros they will probably want this anyway.

Speed, Quality, Size, Cost

Pick 3

[rpurinton]

That's unexpected, TBH. How much faster exactly? I wouldn't mind seeing the encode logs for both (same file, same settings, ram disk vs. SSD).

At first glance, the Transcode RamDisk to RamDisk was running about 39 fps, where SSD to SSD was about 34 fps. I'll run some more tests and share the logs.

Re: RAM usage, heres what I see in top:





And some findings:
Yesterday's test with x264 medium and decomb=fast it took 74 minutes to make all the proxies on that 68 minute 4k video (SSD to SSD)
Today's test with x264 veryfast and decomb not specified going ramdisk to ramdisk it took 51 minutes (Ramdisk to Ramdisk). It took about 15 extra seconds to copy the 7.8GB source file to the Ramdisk first and about 5 seconds to copy back the final proxies to SSD...

See if the veryfast preset is acceptable for you. This is 3x the speed of medium, and often gives a small size that only veryslow can consistently beat. (veryfast uses trellis 0, and medium trellis 1, so it will look a little different.)

It looks like the difference of medium to veryfast is that veryfast used about half as much CPU, however the FPS of the encoding didn't change much (from 34 to 39), it just used less CPU... which is still OK because it means we can transcode more files simultaneously, increasing our total rendering fps...

While I don't know your business model, using veryfast will speed things up at a cost of filesize and/or quality. Which if you are streaming cant be good.

I guess an option might be to allow your users to tweak some of the settings. If they are pros they will probably want this anyway.

Speed, Quality, Size, Cost

Pick 3

They are pro's and most should already be familiar with this information, but we would like to offer custom transcodes instead of trying to find one set of settings that work for everyone.

[tlindgren]

We run 4 simultaneous encodings per submitted video, and sometimes users submit multiple videos simultaneuosly, so we may be running 4,8,12,16,20... HandBrakeCLI's at once.

CentOS 6 was chosen simply because all of our other hundreds of servers are on it, and I feel 99% comfortable with it, whereas CentOS 7, I only feel about 30% confident supporting it... they changed so many things and I just haven't figured out all the differences... If you know a good guide for converting C6 admins into C7 admins I'd happily try it out, if you think this will boost our encoder performance.

If you end up running too many concurrent encodes you can start wasting enough time on task switching that the total throughput starts to decrease, so you probably want to implement a limit on the maximum number of concurrent encodes and a backlog queue. atop is a good tool and it should be easily visible if you start task thrashing (CPU "sys" hitting 1000%+ in your case).

Without testing on your actual hardware it's hard to say how high that number should be and it also depends on the resolution and profile of each encode (lower resolution and faster profiles tend to use less HW) is being done in but for throughput optimization I'd guess it's likely to be somewhere in the 12-24 range for that 4x4850v2 and 2-4 range for that E3-1270v3. Lower numbers likely give faster encodes but lower max throughput (less efficient HW usage).

As long as you can compile the tools you need you should be fine with using CentOS 6, not needing the GUI helps a lot.

RHEL6 is supported to 30 November 2020 (EoPP), while CentOS doesn't promise to provide new versions/updates to that date it's fairly likely they will. RHEL7 EoPP is currently at June 30, 2024 so C7 would likely give you 3.5+ years longer before having to make OS changes, but has to be weighted against having to learn a new OS (coming from C6 there's many really nice parts, and some really weird parts).

I do remember that there was some kernel schedule changes that helped x264 significantly at some point but I have no idea if the standard C6 kernel would be affected by that or not...

But there's no need to change to C7 for just that, elrepo has already done the work providing two different much newer kernels that works with CentOS 6 (kernel-ml is currently at 4.4.0, kernel-lt is at 3.10.95).

Since you're obviously going to run a lot of encodes I suspect it's worth testing at least the latest (kernel-ml), if you don't see significant benefits you can go back to the stock C6 kernel (*heavily* modified 2.6.32), it's even possible the older kernel is faster. Using kernel-ml or kernel-lt means you're on a bit less tested ground so personally I'd only use it if benchmarking says it helps.

[rollin_eng]

They are pro's and most should already be familiar with this information, but we would like to offer custom transcodes instead of trying to find one set of settings that work for everyone.

I think we are saying the same thing here

If they are allowed to customize their settings you just need as many cpu/MHz as you need.

[rpurinton]

I do remember that there was some kernel schedule changes that helped x264 significantly at some point but I have no idea if the standard C6 kernel would be affected by that or not...

But there's no need to change to C7 for just that, elrepo has already done the work providing two different much newer kernels that works with CentOS 6 (kernel-ml is currently at 4.4.0, kernel-lt is at 3.10.95).

Since you're obviously going to run a lot of encodes I suspect it's worth testing at least the latest (kernel-ml), if you don't see significant benefits you can go back to the stock C6 kernel (*heavily* modified 2.6.32), it's even possible the older kernel is faster. Using kernel-ml or kernel-lt means you're on a bit less tested ground so personally I'd only use it if benchmarking says it helps.

Sounds good, since we're in testing now, I've gone ahead and updated to the 4.4 kernel, and will retest!

Also an extra question regarding BIOS settings... Should I test all the different power configs, or is there a generally agreed upon best setting for CPU Power Management, ie: best with Turbo Boost and C-States and stuff, or better to set for Max performance (ie: 100% clock all the time). I'm not concerned about the power draw on the server.

As a side note, for now our management has decided we dont want to make any changes on the quality settings we have been using so far, so we shall be keeping x264 medium, decomb fast, and quality 22.

I'm going to retest as well using RamDisk, SSD, and network storage to see if there is a discernible difference now.

Thanks for all the help so far guys and gals!

[rpurinton]

Looks like using x264 medium with the 4k input about doubled the memory usage versus the previous top GIF

[s55]

HandBrake is not heavy on IO. SSD vs Ramdisk won't make much difference.

HandBrake is heavy on memory bandwidth, not memory usage. Your board probably has high memory bandwidth being a quad xeon, so it may be a non issue with your current settings. Top doesn't show you memory bandwidth usage i believe. The only way you'd know you had a problem is if you spawned more processes and didn't get an increase in throughput. If that does happen, you basically can only fix that by having multiple boxes rather than one large box.

decomb == selective de-interlacing. https://trac.handbrake.fr/wiki/Decomb https://trac.handbrake.fr/wiki/DeinterlacingGuide

[rpurinton]

HandBrake is not heavy on IO. SSD vs Ramdisk won't make much difference.

Yep. Those earlier results must have been some fluke, because after multiple tests now, it makes no difference on Ramdisk or SSD...

To make things even easier for me, I've just tested using SSHFS to read and write directly from the file server (across a WAN with 13ms latency) and even then, I don't see any change, so that saves some time not having to copy the source file over and then copy the proxies back.

[tlindgren]

Also an extra question regarding BIOS settings... Should I test all the different power configs, or is there a generally agreed upon best setting for CPU Power Management, ie: best with Turbo Boost and C-States and stuff, or better to set for Max performance (ie: 100% clock all the time). I'm not concerned about the power draw on the server.

"Max performance" is much easier to benchmark but the power usage will go up. Which in turn means it will be harder to reach the higher Turbo states (due to power or temperature limits) that gives maximum performance...

In my experience "OS Control with C1E" (usually the default) usually gives good performance unless you have special latency requirements, extremely heavy network traffic (10's of gigabits) or are hitting a BIOS bug. The first two shouldn't apply here and the third should hopefully be very unlikely.

Basically, Max Performance is more "robust" to BIOS/OS/latency issues, while "OS Control+C1E" uses less power and can give slightly better max throughput, noticeably better performance at lower load and lower power usage.