Obscene Handbrake Speeds *g*

General questions or discussion about HandBrake, Video and/or audio transcoding, trends etc.
Post Reply
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Obscene Handbrake Speeds *g*

Post by Cyander »

Okay, so I have somewhere around 12+ TV series I still need to import into iTunes from my collection. 0.9.2 with VFR has got me interested in ripping these TV series that are actually variable framerate and I wrote off awhile ago. So, I wrote a couple scripts for the CLI, loop in AtomicParsley to tag my rips immediately after the encode automagically... and I also upgraded my Rev A Mac Pro to use 2.66Ghz Clovertowns. So here is what an 8-way 2.66Ghz, 8GB Mac Pro is pulling down:

2 simultaneous encodes use over 750% CPU using x264 as the encoder. Wow.
A 26-episode half-hour series using a tweaked iPhone High Res preset (but outputting H.264) takes under 4 hours to encode using 2-pass.
Turbo-Pass is producing 250fps avg (300+ peak), while the second pass is producing 125fps avg (180 peak). I would like to see Turbo.264 pull those sorts of numbers out of its hat. :lol:

Just... obscene. And it makes my task of ripping over half a TB of DVDs seem pretty reasonable, rather than daunting as heck.
dynaflash
Veteran User
Posts: 3820
Joined: Thu Nov 02, 2006 8:19 pm

Re: Obscene Handbrake Speeds *g*

Post by dynaflash »

Thats just sick :)
nightstrm
Veteran User
Posts: 1887
Joined: Fri Mar 23, 2007 5:43 am

Re: Obscene Handbrake Speeds *g*

Post by nightstrm »

Cyander wrote:Okay, so I have somewhere around 12+ TV series I still need to import into iTunes from my collection. 0.9.2 with VFR has got me interested in ripping these TV series that are actually variable framerate and I wrote off awhile ago. So, I wrote a couple scripts for the CLI, loop in AtomicParsley to tag my rips immediately after the encode automagically... and I also upgraded my Rev A Mac Pro to use 2.66Ghz Clovertowns. So here is what an 8-way 2.66Ghz, 8GB Mac Pro is pulling down:

2 simultaneous encodes use over 750% CPU using x264 as the encoder. Wow.
A 26-episode half-hour series using a tweaked iPhone High Res preset (but outputting H.264) takes under 4 hours to encode using 2-pass.
Turbo-Pass is producing 250fps avg (300+ peak), while the second pass is producing 125fps avg (180 peak). I would like to see Turbo.264 pull those sorts of numbers out of its hat. :lol:

Just... obscene. And it makes my task of ripping over half a TB of DVDs seem pretty reasonable, rather than daunting as heck.
Great, now I feel compelled to buy an 8-core Mac Pro with 8GB of RAM... do you work for Apple sales or something? :lol:
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Re: Obscene Handbrake Speeds *g*

Post by Cyander »

:lol:

Actually, a Harpertown Mac Pro should be able to get even better performance than I am getting. I went with buying a couple Clovertown CPUs for my Rev A since I found a really cheap source for them that made it make sense over buying a new one.

Oh, and I stand slightly corrected on my own numbers... a 26-episode TV series should take about 3 hours 15 minutes, give or take a few minutes.
mike693
Posts: 46
Joined: Wed Feb 14, 2007 5:42 am

Re: Obscene Handbrake Speeds *g*

Post by mike693 »

Cyander,

When I finally get my Mac Pro, I want to optimize its configuration for HB. How much RAM is in use while the two encodes are running?

Thank you
mike693
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Re: Obscene Handbrake Speeds *g*

Post by Cyander »

That depends. It shouldn't need more than a couple hundred MB at any one time (it seems to stabilize at about 200MB per instance of HB).

In my experience, 0.9.2 has a memory leak which can lead to crashes, but even then, HB only has ~200MB active. So having more RAM won't make HB any faster if it has the 200-300MB free it likes when encoding.
mike693
Posts: 46
Joined: Wed Feb 14, 2007 5:42 am

Re: Obscene Handbrake Speeds *g*

Post by mike693 »

Thanks for the reply. I guess I meant to ask how much total memory is used when you run HB.

I have observed Inactive memory use increases significantly during HandBrake use, until a minimum of Active memory remains. Limited Active memory leads to swapping. I understand this is not HB itself. I suspect this is the filesystem cacheing the VOBs as they are read in.

I figure that, assuming Mac OS X processes can use up to 2GB each, I would need about 2GB for filesystem cache (Finder?), about 500MB for other OS overhead, and then 300MB per instance of HB. (I can't really test this as I am maxed out at 2GB on my Mac Mini.)

Does your Mac use less than 4GB when running two instances of HB?
rhester
Veteran User
Posts: 2888
Joined: Tue Apr 18, 2006 10:24 pm

Re: Obscene Handbrake Speeds *g*

Post by rhester »

Why would you swap is the OS is using the memory as cache? It should convert cache to app memory dynamically as required and only swap if cache approaches zero and you're out of physical free memory.

Have you actually seen this swapping occur?

Rodney
bunny
Posts: 5
Joined: Mon Mar 31, 2008 9:39 pm

Re: Obscene Handbrake Speeds *g*

Post by bunny »

I rip four half hour episodes an hour as 1500kbps deinterlace Xvid and I have a dual core 2.13GHz. Somehow I don't feel I'm missing out on much although I am considering a quad core processor. I would expect you to have got better times unless H.264 adds a huge amount of overhead.
User avatar
s55
HandBrake Team
Posts: 10350
Joined: Sun Dec 24, 2006 1:05 pm

Re: Obscene Handbrake Speeds *g*

Post by s55 »

xVid will see more improvement from clock speed rather than more cores. it's not well multi-threaded so in total, handbrake only uses 1 to 1.5 cores. Same with ffmpeg.
Quad core isn't really the best option unless you like encoding into h.264 (using x264 which works really well on quad core processors)
mike693
Posts: 46
Joined: Wed Feb 14, 2007 5:42 am

Re: Obscene Handbrake Speeds *g*

Post by mike693 »

rhester wrote:Why would you swap is the OS is using the memory as cache? It should convert cache to app memory dynamically as required and only swap if cache approaches zero and you're out of physical free memory.

Have you actually seen this swapping occur?

Rodney
This is what I observed after a fresh reboot and starting a new CLI encode. No other apps were running except activity monitor. As the encode progresses, Inactive memory steadily grows until 1.98 of 2GB of total memory is used. At around the 1.9GB mark, page outs start to grow from zero. At the end of the encode, page outs were 59MB.

I was just curious if the OS will cache more than 2GB for the VOBs HB uses as input, and if running HB alone will consume all free memory on a 4GB machine.
rhester
Veteran User
Posts: 2888
Joined: Tue Apr 18, 2006 10:24 pm

Re: Obscene Handbrake Speeds *g*

Post by rhester »

Again, I don't believe this is HandBrake memory usage - it tells you at the end of the encode in the Activity Log precisely how many bytes it allocated. I believe you are looking at disk caching, and observing the OS swapping out parts of itself to favor the cache for performance. This has nothing to do with HandBrake, it is how virtually all Un*x kernels manage memory to maximize performance - any memory not needed by an application will go to disk cache.

Rodney
bunny
Posts: 5
Joined: Mon Mar 31, 2008 9:39 pm

Re: Obscene Handbrake Speeds *g*

Post by bunny »

sr55 wrote:xVid will see more improvement from clock speed rather than more cores. it's not well multi-threaded so in total, handbrake only uses 1 to 1.5 cores. Same with ffmpeg.
Quad core isn't really the best option unless you like encoding into h.264 (using x264 which works really well on quad core processors)
I run multiple encodes at the same time because I noticed that it wasn't well multi-threaded. If I went to quad core I would just run four encodes at the same time. That way I should expect twice the encode rate which would really be useful. I've tried playing with H.264 but the difference in output doesn't justify the extra time encoding it and I only watch the videos on my PC.
Cyander
Experienced
Posts: 94
Joined: Tue Mar 20, 2007 9:19 pm

Re: Obscene Handbrake Speeds *g*

Post by Cyander »

rhester wrote:Again, I don't believe this is HandBrake memory usage - it tells you at the end of the encode in the Activity Log precisely how many bytes it allocated. I believe you are looking at disk caching, and observing the OS swapping out parts of itself to favor the cache for performance. This has nothing to do with HandBrake, it is how virtually all Un*x kernels manage memory to maximize performance - any memory not needed by an application will go to disk cache.

Rodney
I have seen memory leaks in HB that will lead it to use 4GB of VM and crash with 0.9.2 on long queues or certain movies with VFR on... You are right that HB shouldn't have growing memory usage (and it shouldn't really exceed 200MB or so), but my experience and analysis with Instruments shows otherwise. I have opened a thread attempting to discuss it in the bugs forum.
iSAWaUFO
Novice
Posts: 50
Joined: Wed Jan 02, 2008 3:45 am

Re: Obscene Handbrake Speeds *g*

Post by iSAWaUFO »

I have the problem with HandBrake 0.9.3 and Leopard 10.5.6
All my RAM is going Inactive. Slows down my machine, though nothing crashes.
This article suggested that disk reads for large media files could be non-cached, which might alleviate the problem.
http://discussions.apple.com/thread.jsp ... &tstart=30
I supposed this problem is really at the core of the library which does the transcoding, as opposed to the GUI.
nate247
Posts: 17
Joined: Wed Aug 22, 2007 9:02 am

Re: Obscene Handbrake Speeds *g*

Post by nate247 »

I have the new Intel Xeon “Nehalem" Mac Pro (2x Quad 2.26GHZ) with 8GB Ram.

Tries using Handbrake to do some file conversions using the ATV preset at 63% CB and every time Handbrake crashes about half way through. Is there something I am doing wrong as the main reason I bought a Mac Pro was to encode my DVD & HD DVD library :( lol
TedJ
Veteran User
Posts: 5388
Joined: Wed Feb 20, 2008 11:25 pm

Re: Obscene Handbrake Speeds *g*

Post by TedJ »

Without any logs, your only option is to send me your Mac Pro so I can perform some exhaustive testing (ahem ;)) at this end.

C'mon Nate, you should know better.
nate247
Posts: 17
Joined: Wed Aug 22, 2007 9:02 am

Re: Obscene Handbrake Speeds *g*

Post by nate247 »

I'll post it out tomorrow ;-) lol

Sorry to hijack the thread slightly. It was more of a tongue in cheek comment in that I just wondered if I was being a complete pleb and not doing something correctly as its been a while since I did any encoding. I currently have one final encode running and will see if that too crashes. If it does then I will start up a new thread here and hopefully you can lend a much appreciate hand TedJ :wink:

On a slight side note, since I was last using Handbrake I must say that the quality of encodes has really stepped up a notch or two. Literally no decernable difference between source and output file even when using inbuilt presets. If I can get my Mac Pro working with it, you have a donation winging its way over to you.

Will keep people posted as I guess I am as keen as you all to see just how quick these new Intel processors are at crunching DVD's & HD stuff. Appears a few here have the previous gen Mac pro's and are getting some great results, so would love to see whats possible with the latest gen.

Encode is being left overnight as only started it a minute or two ago, so will post in the morning. Many thanks, Nathan :D

EDIT: It just crashed again about a third in and only getting about 30/40 fps on encode (not entirely sure if thats good or not as not as up to speed with all of this as I would like to be). Will start new thread now though and sorry to OP for slight tangent I have taken the thread. Thanks again.
dynaflash
Veteran User
Posts: 3820
Joined: Thu Nov 02, 2006 8:19 pm

Re: Obscene Handbrake Speeds *g*

Post by dynaflash »

nate247 wrote: If I can get my Mac Pro working with it, you have a donation winging its way over to you.
while we appreciate the thought. as know we do not accept donations.
Da Man
Bright Spark User
Posts: 181
Joined: Sat Mar 08, 2008 5:47 pm

Re: Obscene Handbrake Speeds *g*

Post by Da Man »

They do take beer and liquor donations, though. Send them to my address, and I will dutifully forward all proceeds in a timely and fair fashion starting with the most polite mods and working from there. :lol:

Now who were the guys working on this program again?
Da Man
Bright Spark User
Posts: 181
Joined: Sat Mar 08, 2008 5:47 pm

Re: Obscene Handbrake Speeds *g*

Post by Da Man »

You know, slightly off topic (but concerning donations), it was mentioned awhile back that as a substitution for donating to the Dev crew, a donation be made to a worth-while international organization. You guys ever give any more thought to that? I have several dollars that will not be going to my usual political party due to complete and utter failure to perform. I'd love to send it elsewhere to something else I derive value and entertainment from... :?:
User avatar
s55
HandBrake Team
Posts: 10350
Joined: Sun Dec 24, 2006 1:05 pm

Re: Obscene Handbrake Speeds *g*

Post by s55 »

The problem with the idea, is deciding where. It's not something that's been thought about much yet.
Da Man
Bright Spark User
Posts: 181
Joined: Sat Mar 08, 2008 5:47 pm

Re: Obscene Handbrake Speeds *g*

Post by Da Man »

I still like the EFF idea. Lately, they've assisted in some precedent setting cases with both the RIAA and MPAA, not to mention wire-tapping and DRM-related issues. I'll shift this over to the already started thread just to keep things organized.
nate247
Posts: 17
Joined: Wed Aug 22, 2007 9:02 am

Re: Obscene Handbrake Speeds *g*

Post by nate247 »

Well I would most certainly be happy to do so if it ever came to fruition... :o

Back on the subject of speed. My Mac Pro (newest 8 core model) is running at about 50/60 frames per second average whilst currently encoding my three series of Battlestar Galactica DVD's. Approx 15/16 mins per 45 minute episode. Have no idea if this is good or not, so looking for you guys to say yes / no :-) lol

CBR 59% being used BTW...
dynaflash
Veteran User
Posts: 3820
Joined: Thu Nov 02, 2006 8:19 pm

Re: Obscene Handbrake Speeds *g*

Post by dynaflash »

nate247 wrote:Back on the subject of speed. My Mac Pro (newest 8 core model) is running at about 50/60 frames per second average whilst currently encoding my three series of Battlestar Galactica DVD's. Approx 15/16 mins per 45 minute episode. Have no idea if this is good or not, so looking for you guys to say yes / no :-) lol

CBR 59% being used BTW...
well, I presume by "CBR" (which would be constant bit rate) you mean Constant Quality. At any rate, the options for x264 would probably mean even more than the "CQ" ;) being used as far as benchmarks are concerned.
Post Reply