video material for benchmarking & testing

Post your testing results with HandBrake.
Post Reply
KonaBlend
Regular User
Posts: 72
Joined: Tue Nov 04, 2008 2:35 am

video material for benchmarking & testing

Post by KonaBlend »

I'm a fan of the often presented idea to use some consistent media (maybe in different formats) for benchmarking purposes.
Here's a short clip found (originally from Vimeo) but also available in 1920x1080 as h264 in a mov container; I contacted Tom of Timescapes.org and asked permission to use the vid for HB bench/test purposes. He's graciously agreed; in the interests of not killing his download server, here's a link to the Vimeo HD version:

http://www.vimeo.com/hd#4038064

Maybe the right steps would be to take the mov/h264, convert it to dvd/mpeg2 as typical source material and maybe some larger mpeg2 streams in 720p and 1080p format.

Thoughts?

hugeman
Posts: 22
Joined: Wed Apr 01, 2009 11:23 am

Re: video material for benchmarking & testing

Post by hugeman »

Yes, benchmarks i totally pointless if we do not use the same video clips with the same settings in a bunch of pre-defined tests. Examples of test could be with the Apple-TV preset:

DVD-clip -> H.264
1080p mkv -> 720p H.264
720p mkv - > 720p H.264
etc... etc...

jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: video material for benchmarking & testing

Post by jbrjake »

Also, be aware that lots of projects already have sample archives we can use.

http://samples.mplayerhq.hu/

http://media.xiph.org/video/

http://x264dev.multimedia.cx/?p=14

KonaBlend
Regular User
Posts: 72
Joined: Tue Nov 04, 2008 2:35 am

Re: video material for benchmarking & testing

Post by KonaBlend »

Here's an initial attempt at some material which may be suitable for comparison benchmarking.
Others can perform encoding benchmarks against the same DVD which is about 240MB in size and split into 5 rar files and available for download from: http://www.4shared.com/dir/14538871/5c1f0079/media.html

movie: Learning To Fly (by Tom @ Timescapes.org)
media: LEARNINGTOFLY.img (DVD image on HDD and volume mounted for benchmark)
size: 243,302,400 bytes (md5sum 25498c1c201db57c4819550e64af0a4a)
duration: 00:04:18
video: 720x480, 16:9, NTSC, 23.976 fps
audio: Dolby, 2 channels, English

BENCHMARK: Mac OS X 10.5.6, iMac (Early 2008) 2.8GHz, 2 cores total
version: HandBrake svn2324 (2009041401) - Darwin i386
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 28.002495 fps
time: 219.44 seconds

BENCHMARK: Mac OS X 10.5.6, iMac (Early 2008) 2.8GHz, 2 cores total
version: HandBrake svn2324 (2009041401) - Darwin x86_64
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 30.258183 fps
time: 203.25 seconds

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2008) 2x2.8GHz, 8 cores total
version: HandBrake svn2324 (2009041401) - Darwin i386
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 98.088257 fps
time: 66.94 seconds

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2008) 2x2.8GHz, 8 cores total
version: HandBrake svn2324 (2009041401) - Darwin x86_64
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 106.285713 fps
time: 59.44 seconds
Last edited by KonaBlend on Tue Apr 14, 2009 7:10 pm, edited 1 time in total.

sasha
Posts: 38
Joined: Mon Jan 29, 2007 3:16 pm

Re: video material for benchmarking & testing

Post by sasha »

Hi KonaBlend,

I have downloaded the 5 rar files but during expanding I get an error "Please locate the file named LEARNINGTOFLYpart1.r00."
There seems to be no such file. Perhaps I'm doing something wrong.

Thank you for setting this up!

KonaBlend
Regular User
Posts: 72
Joined: Tue Nov 04, 2008 2:35 am

Re: video material for benchmarking & testing

Post by KonaBlend »

I just use this command; the trick is to tell your program to unpack the first file and if it understands rar then it should know how to glob for the rest...

Code: Select all

unrar x LEARNINGTOFLY.part1.rar
EDIT* ah the download mechanism has borked the filenames! Try renaming your downloaded files so they have two periods as shown above.

sasha
Posts: 38
Joined: Mon Jan 29, 2007 3:16 pm

Re: video material for benchmarking & testing

Post by sasha »

Hi KonaBlend worked as explained.

Below are the results

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2009) 2x2.26GHz, 8 cores total
version: HandBrake svn2314 (2009041001) - x86_64
Source: Learning to fly
preset AppleTV
average encoding: 98.574272 fps

Great test!

hugeman
Posts: 22
Joined: Wed Apr 01, 2009 11:23 am

Re: video material for benchmarking & testing

Post by hugeman »

Perfect KonaBlend. Just what we needed, but in my opinion, I think that we should use the latest official version of the HandBrake in our tests. It gives the most fair values if we use the same version because different versions may have different
optimizations. Or is there any aspect that makes a compilation a must?

But it is always fun to see how many fps you can get with various compilations of SVN versions =)

KonaBlend
Regular User
Posts: 72
Joined: Tue Nov 04, 2008 2:35 am

Re: video material for benchmarking & testing

Post by KonaBlend »

Yes, official releases are sound advice for benchmarking. But there are no quick and fast rules; ie: it's valid to compare 0.9.3 against trunk too. Others are giving advice on which options to use for encoding; and I think i found a filehost that won't mangle filenames so we're getting close to a good base. Not there yet tho.

jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: video material for benchmarking & testing

Post by jbrjake »

hugeman wrote:I think that we should use the latest official version of the HandBrake in our tests. It gives the most fair values if we use the same version because different versions may have different optimizations.
People just need to indicate what version and architecture they're using the way Kona and sasha are...

KonaBlend
Regular User
Posts: 72
Joined: Tue Nov 04, 2008 2:35 am

Re: video material for benchmarking & testing

Post by KonaBlend »

I've written a python script which automates running and recording the results of a benchmark. It's still a work in progress but here's a sample result output from HandBrake 0.9.3 CLI (no GUI support).

Code: Select all

date: Fri Apr 17 11:32:29 2009
system: Mac OS X 10.5.6 (9G55) | MacPro3,1 | Quad-Core Intel Xeon | 2 x 2.8 GHz (total cores: 8)
handbrake: HandBrake 0.9.3 (2008112300) (7590df4f49c3585ce691455e5f0745b0)
media: LearningToFly-NTSC@24p.m2t (f3f357f857b711117b14c38053ace08a)
benchmark: 101.ntsc - x264 LearningToFly NTSC@24p
encoding rate: 132.228012 fps
elapsed time: 48.07 seconds
cpu time: 360.62 seconds (750.26%)
args: --verbose --input ./LearningToFly-NTSC@24p.m2t --output out-LearningToFly-NTSC@24p.m4v --encoder x264 --quality 20 --pixelratio
Any comments on the command-line args used? Or potential other information which might be useful to record for benchmark posterity?

cheerful
Posts: 7
Joined: Fri Apr 25, 2008 4:51 pm

Re: video material for benchmarking & testing

Post by cheerful »

KonaBlend, that's a nice touch. As much as love to play with CLI but I'm still a fan of UI ;)

hugeman
Posts: 22
Joined: Wed Apr 01, 2009 11:23 am

Re: video material for benchmarking & testing

Post by hugeman »

Just got my new Mac Pro Quad 2.66 GHz (Nehalem), I ran some quick tests and compared to my old computer

BENCHMARK: Mac OS X 10.5.6, iMac (24" white) 2.16GHz, 2 cores total
version: HandBrake 0.93
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 18.011303

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Nehalem) 2.66GHz, 4 cores total
version: HandBrake 0.93
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 74.845856

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Nehalem) 2.66GHz, 4 cores total
version: HandBrake SVN 2412 x86_64 (4 instances)
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 85.964890 fps

I will update later with some svn-compilation tests.

Post Reply