Page 1 of 1

video material for benchmarking & testing

Posted: Mon Apr 13, 2009 1:54 am
by KonaBlend
I'm a fan of the often presented idea to use some consistent media (maybe in different formats) for benchmarking purposes.
Here's a short clip found (originally from Vimeo) but also available in 1920x1080 as h264 in a mov container; I contacted Tom of Timescapes.org and asked permission to use the vid for HB bench/test purposes. He's graciously agreed; in the interests of not killing his download server, here's a link to the Vimeo HD version:

http://www.vimeo.com/hd#4038064

Maybe the right steps would be to take the mov/h264, convert it to dvd/mpeg2 as typical source material and maybe some larger mpeg2 streams in 720p and 1080p format.

Thoughts?

Re: video material for benchmarking & testing

Posted: Mon Apr 13, 2009 11:24 am
by hugeman
Yes, benchmarks i totally pointless if we do not use the same video clips with the same settings in a bunch of pre-defined tests. Examples of test could be with the Apple-TV preset:

DVD-clip -> H.264
1080p mkv -> 720p H.264
720p mkv - > 720p H.264
etc... etc...

Re: video material for benchmarking & testing

Posted: Mon Apr 13, 2009 2:33 pm
by jbrjake
Also, be aware that lots of projects already have sample archives we can use.

http://samples.mplayerhq.hu/

http://media.xiph.org/video/

http://x264dev.multimedia.cx/?p=14

Re: video material for benchmarking & testing

Posted: Mon Apr 13, 2009 8:53 pm
by KonaBlend
Here's an initial attempt at some material which may be suitable for comparison benchmarking.
Others can perform encoding benchmarks against the same DVD which is about 240MB in size and split into 5 rar files and available for download from: http://www.4shared.com/dir/14538871/5c1f0079/media.html

movie: Learning To Fly (by Tom @ Timescapes.org)
media: LEARNINGTOFLY.img (DVD image on HDD and volume mounted for benchmark)
size: 243,302,400 bytes (md5sum 25498c1c201db57c4819550e64af0a4a)
duration: 00:04:18
video: 720x480, 16:9, NTSC, 23.976 fps
audio: Dolby, 2 channels, English

BENCHMARK: Mac OS X 10.5.6, iMac (Early 2008) 2.8GHz, 2 cores total
version: HandBrake svn2324 (2009041401) - Darwin i386
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 28.002495 fps
time: 219.44 seconds

BENCHMARK: Mac OS X 10.5.6, iMac (Early 2008) 2.8GHz, 2 cores total
version: HandBrake svn2324 (2009041401) - Darwin x86_64
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 30.258183 fps
time: 203.25 seconds

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2008) 2x2.8GHz, 8 cores total
version: HandBrake svn2324 (2009041401) - Darwin i386
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 98.088257 fps
time: 66.94 seconds

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2008) 2x2.8GHz, 8 cores total
version: HandBrake svn2324 (2009041401) - Darwin x86_64
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 106.285713 fps
time: 59.44 seconds

Re: video material for benchmarking & testing

Posted: Tue Apr 14, 2009 6:58 pm
by sasha
Hi KonaBlend,

I have downloaded the 5 rar files but during expanding I get an error "Please locate the file named LEARNINGTOFLYpart1.r00."
There seems to be no such file. Perhaps I'm doing something wrong.

Thank you for setting this up!

Re: video material for benchmarking & testing

Posted: Tue Apr 14, 2009 7:15 pm
by KonaBlend
I just use this command; the trick is to tell your program to unpack the first file and if it understands rar then it should know how to glob for the rest...

Code: Select all

unrar x LEARNINGTOFLY.part1.rar
EDIT* ah the download mechanism has borked the filenames! Try renaming your downloaded files so they have two periods as shown above.

Re: video material for benchmarking & testing

Posted: Tue Apr 14, 2009 8:27 pm
by sasha
Hi KonaBlend worked as explained.

Below are the results

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Early 2009) 2x2.26GHz, 8 cores total
version: HandBrake svn2314 (2009041001) - x86_64
Source: Learning to fly
preset AppleTV
average encoding: 98.574272 fps

Great test!

Re: video material for benchmarking & testing

Posted: Wed Apr 15, 2009 4:31 pm
by hugeman
Perfect KonaBlend. Just what we needed, but in my opinion, I think that we should use the latest official version of the HandBrake in our tests. It gives the most fair values if we use the same version because different versions may have different
optimizations. Or is there any aspect that makes a compilation a must?

But it is always fun to see how many fps you can get with various compilations of SVN versions =)

Re: video material for benchmarking & testing

Posted: Wed Apr 15, 2009 5:03 pm
by KonaBlend
Yes, official releases are sound advice for benchmarking. But there are no quick and fast rules; ie: it's valid to compare 0.9.3 against trunk too. Others are giving advice on which options to use for encoding; and I think i found a filehost that won't mangle filenames so we're getting close to a good base. Not there yet tho.

Re: video material for benchmarking & testing

Posted: Wed Apr 15, 2009 6:03 pm
by jbrjake
hugeman wrote:I think that we should use the latest official version of the HandBrake in our tests. It gives the most fair values if we use the same version because different versions may have different optimizations.
People just need to indicate what version and architecture they're using the way Kona and sasha are...

Re: video material for benchmarking & testing

Posted: Fri Apr 17, 2009 3:44 pm
by KonaBlend
I've written a python script which automates running and recording the results of a benchmark. It's still a work in progress but here's a sample result output from HandBrake 0.9.3 CLI (no GUI support).

Code: Select all

date: Fri Apr 17 11:32:29 2009
system: Mac OS X 10.5.6 (9G55) | MacPro3,1 | Quad-Core Intel Xeon | 2 x 2.8 GHz (total cores: 8)
handbrake: HandBrake 0.9.3 (2008112300) (7590df4f49c3585ce691455e5f0745b0)
media: LearningToFly-NTSC@24p.m2t (f3f357f857b711117b14c38053ace08a)
benchmark: 101.ntsc - x264 LearningToFly NTSC@24p
encoding rate: 132.228012 fps
elapsed time: 48.07 seconds
cpu time: 360.62 seconds (750.26%)
args: --verbose --input ./LearningToFly-NTSC@24p.m2t --output out-LearningToFly-NTSC@24p.m4v --encoder x264 --quality 20 --pixelratio
Any comments on the command-line args used? Or potential other information which might be useful to record for benchmark posterity?

Re: video material for benchmarking & testing

Posted: Tue Apr 21, 2009 8:06 am
by cheerful
KonaBlend, that's a nice touch. As much as love to play with CLI but I'm still a fan of UI ;)

Re: video material for benchmarking & testing

Posted: Mon May 11, 2009 5:16 pm
by hugeman
Just got my new Mac Pro Quad 2.66 GHz (Nehalem), I ran some quick tests and compared to my old computer

BENCHMARK: Mac OS X 10.5.6, iMac (24" white) 2.16GHz, 2 cores total
version: HandBrake 0.93
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 18.011303

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Nehalem) 2.66GHz, 4 cores total
version: HandBrake 0.93
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 74.845856

BENCHMARK: Mac OS X 10.5.6, Mac Pro (Nehalem) 2.66GHz, 4 cores total
version: HandBrake SVN 2412 x86_64 (4 instances)
command: ./HandBrakeCLI --verbose --input /Volumes/LEARNINGTOFLY --output fly.m4v --preset AppleTV
average encoding: 85.964890 fps

I will update later with some svn-compilation tests.