2GB Limit?

Polygon · Post by **Polygon** » Mon Nov 26, 2007 1:43 am

ill try a precompiled binary and get back to you

redraiderbum · Post by **redraiderbum** » Mon Nov 26, 2007 5:43 pm

Are you, by any chance, using 32-bit ubuntu on a 64-bit processor? I've had file size limit problems regardless of the filesystem doing this. It may come from handbrake detecting a 64-bit processor for compiling, then the OS being limited to manipulating 32-bit blocks of data even though handbrake can do bigger... I'm not sure.

Also, just fyi, even if you're using a filesystem that is beyond 32-bit, if the OS is working with a particular file and the OS is 32-bit you can run into problems because the OS can't address the entire file at the same time; in addition, although the amount of memory addressable shouldn't be limited (assuming a 64-bit processor), I've run into problems with the size of the swap on 32-bit systems. I'm not sure what the technical reasons are for this, but I have had these specific problems on various systems.

That said, I thought the size limit for 32 bits was 4 GB per block? I was under the impression fat32 was limited to 2GB for other reasons?

keller · Post by **keller** » Mon Nov 26, 2007 6:08 pm

Yes, I am running 32-bit Ubuntu on a 64-bit AMD Dual Core proc. Did this same issue exist in previous version of Ubuntu, or is it new to Gutsy?

Anyone know how I can force HandBrake to detect my processor as a 32-bit one?

-Keller

Polygon · Post by **Polygon** » Mon Nov 26, 2007 8:10 pm

redraiderbum wrote:Are you, by any chance, using 32-bit ubuntu on a 64-bit processor? I've had file size limit problems regardless of the filesystem doing this. It may come from handbrake detecting a 64-bit processor for compiling, then the OS being limited to manipulating 32-bit blocks of data even though handbrake can do bigger... I'm not sure.

Also, just fyi, even if you're using a filesystem that is beyond 32-bit, if the OS is working with a particular file and the OS is 32-bit you can run into problems because the OS can't address the entire file at the same time; in addition, although the amount of memory addressable shouldn't be limited (assuming a 64-bit processor), I've run into problems with the size of the swap on 32-bit systems. I'm not sure what the technical reasons are for this, but I have had these specific problems on various systems.

That said, I thought the size limit for 32 bits was 4 GB per block? I was under the impression fat32 was limited to 2GB for other reasons?

I am running an AMD Athlon XP 3000+ 2.1 ghz, which is a 32 bit processor.

and i believe the file size limit of fat 32 is 4 gb, and the 4 gb thing your bringing up with 32 bit systems is how much memory (ram) can be addressed by a 32 bit system, not file size.

and my rip is just about to hit the 2.0 gb mark, lets see if it dumps core.

EDIT: yep, did it with the precompiled binary too.

mark@Cactus-Fantastico:~$ cd Desktop/Handbrake_091/
mark@Cactus-Fantastico:~/Desktop/Handbrake_091$ md5sum HandBrakeCLI
4e2724ba5294f72210bb4be48c07fae7 HandBrakeCLI
mark@Cactus-Fantastico:~/Desktop/Handbrake_091$ ./HandBrakeCLI -i /media/cdrom0/ -o /media/LinuxBackup/Ripped\ DVDs/The_Lord_of_the_Rings_1_-_The_Fellowship_of_the_Ring_x264.mkv -e x264 -b 1600 -B 160 -R 48 -E ac3 -f mkv -x ref=5:mixed-refs:bframes=3:bime:weightb:b-rdo:b-pyramid:me=umh:subme=7:trellis=1:analyse=all:8x8dct:no-fast-pskip -m -p -2 -T
HandBrake 0.9.1 (2007100800) - http://handbrake.m0k.org/
1 CPU detected
Opening /media/cdrom0/...
Scanning title 1...
+ title 1:
+ vts 1, ttn 1, cells 0->40 (3663207 blocks)
+ duration: 02:58:10
+ size: 720x480, aspect: 1.78, 23.976 fps
+ autocrop: 58/64/0/0
+ chapters:
+ 1: cells 0->0, 143003 blocks, duration 00:07:15
+ 2: cells 1->1, 104429 blocks, duration 00:04:19
+ 3: cells 2->2, 79015 blocks, duration 00:04:04
+ 4: cells 3->3, 103377 blocks, duration 00:04:23
+ 5: cells 4->4, 84137 blocks, duration 00:04:24
+ 6: cells 5->5, 44819 blocks, duration 00:02:13
+ 7: cells 6->6, 56802 blocks, duration 00:02:41
+ 8: cells 7->7, 182631 blocks, duration 00:09:07
+ 9: cells 8->8, 99856 blocks, duration 00:04:37
+ 10: cells 9->9, 90205 blocks, duration 00:03:50
+ 11: cells 10->10, 47623 blocks, duration 00:02:17
+ 12: cells 11->11, 109018 blocks, duration 00:05:53
+ 13: cells 12->12, 66588 blocks, duration 00:03:24
+ 14: cells 13->13, 38735 blocks, duration 00:01:42
+ 15: cells 14->14, 98192 blocks, duration 00:04:39
+ 16: cells 15->15, 53665 blocks, duration 00:02:31
+ 17: cells 16->16, 139243 blocks, duration 00:06:12
+ 18: cells 17->17, 38732 blocks, duration 00:02:00
+ 19: cells 18->18, 67671 blocks, duration 00:03:29
+ 20: cells 19->19, 70323 blocks, duration 00:03:33
+ 21: cells 20->20, 36828 blocks, duration 00:02:11
+ 22: cells 21->21, 26611 blocks, duration 00:01:38
+ 23: cells 22->22, 129916 blocks, duration 00:06:57
+ 24: cells 23->23, 35954 blocks, duration 00:01:53
+ 25: cells 24->24, 66725 blocks, duration 00:02:59
+ 26: cells 25->25, 109602 blocks, duration 00:04:41
+ 27: cells 26->26, 101476 blocks, duration 00:04:48
+ 28: cells 27->27, 87715 blocks, duration 00:04:54
+ 29: cells 28->28, 188912 blocks, duration 00:08:57
+ 30: cells 29->29, 208296 blocks, duration 00:09:41
+ 31: cells 30->30, 124235 blocks, duration 00:06:26
+ 32: cells 31->31, 104945 blocks, duration 00:05:28
+ 33: cells 32->32, 37846 blocks, duration 00:01:40
+ 34: cells 33->33, 16017 blocks, duration 00:00:52
+ 35: cells 34->34, 63835 blocks, duration 00:02:44
+ 36: cells 35->35, 134032 blocks, duration 00:06:30
+ 37: cells 36->36, 162533 blocks, duration 00:06:40
+ 38: cells 37->37, 46861 blocks, duration 00:02:34
+ 39: cells 38->38, 136386 blocks, duration 00:06:18
+ 40: cells 39->40, 126418 blocks, duration 00:07:40
+ audio tracks:
+ 1, English (AC3) (5.1 ch), 48000Hz, 448000bps
+ 2, English (AC3) (Dolby Surround), 48000Hz, 192000bps
+ subtitle tracks:
+ 1, English (iso639-2: eng)
Applying the following x264 options: ref=5:mixed-refs:bframes=3:bime:weightb:b-rdo:b-pyramid:me=umh:subme=7:trellis=1:analyse=all:8x8dct:no-fast-pskip
Modified x264 options for pass 1 to append turbo options: ref=5:mixed-refs:bframes=3:bime:weightb:b-rdo:b-pyramid:me=umh:subme=7:trellis=1:analyse=all:8x8dct:no-fast-pskip:ref=1:subme=1:me=dia:analyse=none:trellis=0:no-fast-pskip=0:8x8dct=0
x264 [warning]: width or height not divisible by 16 (720x358), compression will suffer.
x264 [info]: using SAR=32/27
x264 [info]: using cpu capabilities: MMX MMXEXT SSE 3DNow!
Encoding: task 1 of 2, 100.00 % (34.18 fps, avg 26.25 fps, ETA 00h00m00s)x264 [info]: slice I:1806 Avg QP:17.37 size: 33786 PSNR Mean Y:46.78 U:50.19 V:50.29 Avg:47.53 Global:46.82
x264 [info]: slice P:105248 Avg QP:19.41 size: 13034 PSNR Mean Y:44.79 U:49.62 V:49.68 Avg:45.68 Global:44.57
x264 [info]: slice B:149278 Avg QP:20.65 size: 4753 PSNR Mean Y:44.34 U:50.09 V:50.24 Avg:45.29 Global:44.16
x264 [info]: mb I I16..4: 21.4% 0.0% 78.6%
x264 [info]: mb P I16..4: 22.6% 0.0% 0.0% P16..4: 65.8% 0.0% 0.0% 0.0% 0.0% skip:11.6%
x264 [info]: mb B I16..4: 1.3% 0.0% 0.0% B16..8: 27.7% 0.0% 0.0% direct:32.9% skip:38.1%
x264 [info]: final ratefactor: 19.78
x264 [info]: SSIM Mean Y:0.9785674
x264 [info]: PSNR Mean Y:44.543 U:49.895 V:50.011 Avg:45.469 Global:44.342 kb/s:1603.10
x264 [warning]: width or height not divisible by 16 (720x358), compression will suffer.
x264 [info]: using SAR=32/27
x264 [info]: using cpu capabilities: MMX MMXEXT SSE 3DNow!
Encoding: task 1 of 2, 100.00 % (34.18 fps, avg 26.25 fps, ETA 00h00m00s)No accelerated IMDCT transform found
Encoding: task 2 of 2, 80.11 % (4.92 fps, avg 4.20 fps, ETA 03h22m23s)File size limit exceeded (core dumped)
mark@Cactus-Fantastico:~/Desktop/Handbrake_091$ cd /media/LinuxBackup/Ripped\ DVDs/
mark@Cactus-Fantastico:/media/LinuxBackup/Ripped DVDs$ ls -a
-rw-r--r-- 1 mark mark 2147483647 2007-11-26 13:35 The_Lord_of_the_Rings_1_-_The_Fellowship_of_the_Ring_x264.mkv

redraiderbum · Post by **redraiderbum** » Tue Nov 27, 2007 5:45 pm

The 32 bit limit concerns the amount of contiguous bits wired at a particular instant so it is a big problem for system memory, but strictly speaking it could be a problem with anything 32 bit. That said, you are right it's probably not the problem since you are running a complete 32 bit system and the problem is at 2 GB. I have experienced the 32 bit OS on 64 bit processor problem on feisty, pclinuxos, and fedora 7. I can't even get opensuse to compile it so I don't know about that one.

I believe I spoke with dynaflash about this before and neither of us knew the answer, but I have seen this 2 GB limit before. The program I use to rip movies in linux is vobcopy and it has a single file size limit of 2 GB, but it has a large file option if you know you are going over. It seems whoever wrote that program knows the reason or at least the fact that there exists a file size limit in some cases. I'm not sure if the option invoke is an option in the application or the system, but if it's a big deal someone could get the source and search around for the option.

I am currently installing the 64 bit gutsy to see if I can figure it out. FYI, I had handbrake running on a leopard hackintosh and it works perfectly; just wanted to let the dev's know in case someone asks.

Edit: Incidentally, I have had a lot of xfs file system issues on the latest linux kernels, I'm not sure what the deal is.

dynaflash · Post by **dynaflash** » Tue Nov 27, 2007 6:05 pm

redraiderbum wrote:I believe I spoke with dynaflash about this before and neither of us knew the answer, but I have seen this 2 GB limit before.

Not sure what you spoke to me about, but no I have never seen any 2 GB limit when it comes to HB. Though, I only use "hackintosh" exclusively

.

redraiderbum · Post by **redraiderbum** » Tue Nov 27, 2007 11:49 pm

Sorry I meant I have seen the 2 GB file size limit in a linux movie ripping/encoding situation before, here is the post:

http://handbrake.m0k.org/forum/viewtopi ... ht=vobcopy

Actually it appears that rhester was the one I discussed it with... the file size limit is in the program vobcopy which I rip with before using handbrake. I have never seen this problem in handbrake, but the reason the vobcopy large file option may exist is for this reason. Just a possibility.

Yes hackintosh is awesome!

Polygon · Post by **Polygon** » Wed Nov 28, 2007 6:21 am

no...i still think its a problem with our linux boxes and or handbrake, cause jbrjake said that he can encode 2gb+ movies file on darwin (mac os x).....so its technically possible, its just that something on linux is not liking it.

redraiderbum · Post by **redraiderbum** » Wed Nov 28, 2007 4:48 pm

I checked my files last night, it turns out in gutsy 64 bit I have encoded Happy Feet with all of those settings except it is mp4 and it has some advanced x264 settings. That file is 2.12 GB.

According to my records I used 64 bit Ubuntu Gutsy fresh install svn (I'm not sure what revision), ext3 fs, AMD 64 X2 3800.

I will try mkv and post again.

Polygon · Post by **Polygon** » Wed Nov 28, 2007 6:51 pm

I just tried it with teh Dues Quad Ex or whatever preset.

Well maybe its just this movie im encoding, tell me the exact settings that you used to rip happy feet (as in the command line argument) as I have the movie as well, and ill see if i can rip it without it dumping core.

redraiderbum · Post by **redraiderbum** » Thu Nov 29, 2007 6:14 pm

HandBrakeCLI -i Movie.vob -o Movie.mp4 -e x264 -q .6 -Q -E faac -x ref=3:mixed-refs=1:bframes=3:b-rdo=1:direct=auto:subme=6:trellis=1:analyse=all:vbv-maxrate=25000:me=umh

Here is my encode line; I'm not positive this line will work because I only recorded my settings and not the line itself so I just built this line. The x264 advanced settings I got from PuzZLeR so I'm quite certain they are the best they can be within reason. Happy Feet ending up being 2.12 GB as I said before, amazingly my power supply fried on my linux box last night so I ordered a new one. I will test the exact encode line when it comes in.

Polygon · Post by **Polygon** » Fri Nov 30, 2007 7:12 am

i ripped happy feet with vobcopy and im encoding it now with the line you gave me (i mirrored it so its a video_ts folder) so i will post back after i see how it goes.

Polygon · Post by **Polygon** » Fri Nov 30, 2007 1:29 pm

well the rip is done, problem is the resulting file is 1.1 gb, not even close to the 2 gb mark...which is interesting.

maybe you forgot some flags on your command.

redraiderbum · Post by **redraiderbum** » Fri Nov 30, 2007 2:42 pm

I may have, just try it at .65 or .7 to ensure a file bigger than 2 gb. I used anamorphic also.

redraiderbum · Post by **redraiderbum** » Wed Dec 05, 2007 3:49 pm

I have discovered that it is apparently an mkv problem. I used almost exactly the settings given with mkv and I got the 2 GB limit, then I changed only the container to mp4 and it worked. I did the test on Ubuntu Gutsy then again on SME Server with the same results. Ubuntu was 64 bit and SME Server is 32 bit. The final mp4 is 2.35 GB. Here is the encode stuff.

First:

./HandBrakeCLI -i EMPIRE_RECORDS/ -L -T -e x264 -q .7 -Q -U -F -E ac3 -a 1 -6 6ch -o "Empire.mkv"

Encoding: task 1 of 2, 100.00 %x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
No accelerated IMDCT transform found
Encoding: task 2 of 2, 86.72 % (39.09 fps, avg 43.26 fps, ETA 00h07m52s)File size limit exceeded

Second:

./HandBrakeCLI -i EMPIRE_RECORDS/ -L -T -e x264 -q .7 -Q -U -F -E ac3 -a 1 -6 6ch -o "Empire1.mp4"

Encoding: task 1 of 2, 100.00 %No accelerated IMDCT transform found
Encoding: task 2 of 2, 100.00 % (61.24 fps, avg 43.07 fps, ETA 00h00m00s)x264 [info]: slice I:1244 Avg QP:14.83 size: 44393 PSNR Mean Y:47.80 U:49.63 V:49.88 Avg:48.35 Global:48.22
x264 [info]: slice P:152521 Avg QP:17.94 size: 13824 PSNR Mean Y:44.52 U:47.41 V:47.70 Avg:45.30 Global:45.08
x264 [info]: mb I I16..4: 13.2% 0.0% 86.8%
x264 [info]: mb P I16..4: 7.2% 0.0% 6.7% P16..4: 38.8% 30.5% 11.7% 0.0% 0.0% skip: 5.1%
x264 [info]: SSIM Mean Y:0.9827239
x264 [info]: PSNR Mean Y:44.550 U:47.429 V:47.714 Avg:45.323 Global:45.098 kb/s:2698.94
Muxing: 0.00 %
Rip done!

Hope this helps. I'm not sure if this implies a HandBrake problem or not?

Polygon · Post by **Polygon** » Thu Dec 06, 2007 6:33 am

as ive ripped the LOTR1 dvd on windows, with the same settings that are generating this '2 gb' errors....maybe its a problem with like the linux version of libmatroska or whatever is used to generate mkv files?

redraiderbum · Post by **redraiderbum** » Thu Dec 06, 2007 3:06 pm

I have no idea, I don't typically use matroska, but at least we know what it is. I'm going to put a thread in the bugs forum to make sure the devs see it; although, I'm not sure if they can do anything about it.

tealcomp · Post by **tealcomp** » Thu Dec 06, 2007 3:33 pm

Just to chime in here; I am seeing the same issue; at first I thought it was the GTK I was using, but then it happened with the CLI as well. I will post the required particulars later, but I am running Ubunutu 7.04 64bit on a Dual Core Centrino. I thought it was just me doing something wrong, glad to see others are also experiencing this.

-Dan

jbrjake · Post by **jbrjake** » Thu Dec 06, 2007 6:00 pm

So in my first post in this thread I told all of you that saintdev would be the one to rule out any libmkv-specific problems, since it's his library.

How come none of you have bothered to get in touch with him, is this bug is so important to you?

tealcomp · Post by **tealcomp** » Thu Dec 06, 2007 9:10 pm

jbrjake wrote:So in my first post in this thread I told all of you that saintdev would be the one to rule out any libmkv-specific problems, since it's his library.

How come none of you have bothered to get in touch with him, is this bug is so important to you?

Not sure if that was directed at me specifically or not; if it was, at this point I am simply sharing information with others of this forum that are experiencing similar problems.

Polygon · Post by **Polygon** » Sat Dec 08, 2007 3:43 pm

jbrjake wrote:So in my first post in this thread I told all of you that saintdev would be the one to rule out any libmkv-specific problems, since it's his library.

How come none of you have bothered to get in touch with him, is this bug is so important to you?

because at least i didn't know that

A: this was a mkv problem (until recently)
B: that saintdev was the expert on libmkv

so i'll pm him and point him to this thread to see if he can give us any valuable input

Polygon · Post by **Polygon** » Tue Dec 11, 2007 4:06 am

It appears to be already fixed, and will go into svn soon

from saintdev:

. I'm pretty sure I already found this and fixed it in my local repo.

saintdev · Post by **saintdev** » Tue Dec 11, 2007 4:11 am

Polygon wrote:It appears to be already fixed, and will go into svn soon

from saintdev:
. I'm pretty sure I already found this and fixed it in my local repo.

Actually no that's not what I meant. I'm not experienceing this so I'm not sure if it IS fixed or not. Plus this would be an issue with libmkv, not with any of the HandBrake code so it won't be going into svn until I update libmkv. Which I've been very much neglecting. I do however need someone to test the updated package for me to see if they still have the problem or not.

If you are able to compile HB on your own, are hitting this bug, and wouldn't mind testing, send me a PM and I'll setup a way for you to see if it's fixed or not.

hki · Post by **hki** » Wed Dec 12, 2007 4:15 pm

saintdev wrote:
Polygon wrote:It appears to be already fixed, and will go into svn soon

from saintdev:
. I'm pretty sure I already found this and fixed it in my local repo.
Actually no that's not what I meant. I'm not experienceing this so I'm not sure if it IS fixed or not. Plus this would be an issue with libmkv, not with any of the HandBrake code so it won't be going into svn until I update libmkv. Which I've been very much neglecting. I do however need someone to test the updated package for me to see if they still have the problem or not.

If you are able to compile HB on your own, are hitting this bug, and wouldn't mind testing, send me a PM and I'll setup a way for you to see if it's fixed or not.

I have compiled version 0.9.1 from the SVN source on 25.10.2007. I am using 64bit version of Ubuntu. Today I tested with some video by bumping the bitrate to 5000 and using x264 codec and ac3 and got a 2,9 GB file in mkv container. So I cannot replicate probem now. BUT, I have seen this error before when encoding on the same system. It must have been before the compile then.

Polygon · Post by **Polygon** » Wed Dec 12, 2007 6:29 pm

saintdev gave me a newer version of libmkv to test out and i just ripped LOTR 2, and the resulting file was 2.6 gb and it encoded successfully without dumping core, so i guess once he commits the changes to svn, it will all be fixed