Action if mp4 without large file size selected gets too big

[rachel]

An annoying thing I'm finding is that if I set a batch of stuff to be converted using eg: Normal or High Profile to mp4 format, by default the "Large file size" option isn't selected. But if a given file grows over the 4GB limit there's no kind of warning that anything's amiss. It just plugs on creating a broken mpeg4.

It's not always straightforward to anticipate when a given transcode will get too big as it's very dependent on the source material.

Is it possible we could have some automatic behaviour to avoid this. eg:

1: Regular->Normal and Regular->High Profile having Large File Size enabled by default. (Yes I know I can make my own presets to do this, but they won't get automatically updated when the standard ones do. Now if user-defined presets could be defined as inheriting from basic ones with the listed changes... but that's another story...)

2: If a file grows too big when Large File Size is not enabled, for it to abandon the encode with a visible error, so the user could Edit the queue item, check the checkbox, and start it again.

3: Or automate 2: If the file gets too big, for handbrake to autonymously abandon it, set that option and restart that encode.

4: Or would it be possible to convert the output so far to large-file-size and resume? Whether in-situ or in an automatic remux-to-new-file-and-resume. After all a lot of codec heavy lifting will have been done and could hopefully just be copied/remuxed across to a new large-file container far more quickly than all that encoding having to be done again.

[rachel]

Just to offer the practical example that triggered this post: After several days my Mac recently completed encoding my blu-ray Planet Earth episodes to mp4 using the High Profile preset. It only manages at best about 5fps encoding speed on this preset... So imagine my frustration when it turned out that 6 of the 11 files produced were broken. They were all *around* 4GB in size, but some just edged over the limit and some didn't. The only visible sign that something was wrong before trying to play them was that the Finder previews weren't working - but sometimes they don't work anyway, such as when the Finder saw the file as a work in progress and remembered it didn't have a preview rather than trying again.

[jbrjake]

Large file size will be the default in 0.9.5.

[mduell]

But if a given file grows over the 4GB limit there's no kind of warning that anything's amiss.

[censored]. Read your activity logs, they clearly say MP4ERROR: (m_size <= (uint64_t)0xFFFFFFFF): assert failure

[rachel]

But if a given file grows over the 4GB limit there's no kind of warning that anything's amiss.

[censored]. Read your activity logs, they clearly say MP4ERROR: (m_size <= (uint64_t)0xFFFFFFFF): assert failure

Nice. I did look at the activity log. If it said that I didn't see it so it's at best not clear. And besides, what the [censored] is the [censored] point of continuing if the file is going to be [censored] from that point on? What's the [censored] point in not reporting it to the [censored] user in some [censored] fashion but carrying on as if everything's fine? Obviously it stops as soon as it finds the error, but it doesn't realise until it's got to the end of a file? There's a [censored] six gigabyte output file here; so it carried on [censored] encoding for another 2GB after it should have realised it was writing a broken file. Which in High Profile on a HD source is a [censored] lot of cycles. So even if I had been so bored as to watch the activity log while it was encoding I wouldn't have seen the error until it was too late anyway, would I? As it was, it was a batch running over several [censored] days and it would be [censored] useful if it had some kind of [censored] sensible default behaviour in case something [censored] happened, wouldn't it?

NB: I'm not in the kind of spitting rage that the above might convey, those [censored]s I wrote as such myself. I'm being sarcastic. This was not a stupid newbie question and jbrjake's response was perfectly adequate, both to the question and to the actual problem, and is the obviously sensible way forward. (Presumably devices that can't handle >4GB mp4s can now be considered "legacy" thank God.) But as it stands it's a fatal error that breaks the encode and demands user attention, and one line buried in the activity log is by no standards an adequate way of handling it. It even returns a success (0) result code! (last line of the log) One question I suppose is why, if that's planned for 0.9.5, the presets on the current SVNs I'm running don't already have large file as the default - but maybe it's being wiped out when I say yes to letting it automatically update my presets from, presumably, what the build provides.

[mduell]

The behavior is completely [censored] sideways, but you can't [censored] say you weren't warned.

[rachel]

The behavior is completely [censored] sideways, but you can't [censored] say you weren't warned.

In the bottom of a locked filing cabinet... you know the rest.

[jbrjake]

And besides, what the [censored] is the [censored] point of continuing if the file is going to be [censored] from that point on? What's the [censored] point in not reporting it to the [censored] user in some [censored] fashion but carrying on as if everything's fine? Obviously it stops as soon as it finds the error, but it doesn't realise until it's got to the end of a file? There's a [censored] six gigabyte output file here; so it carried on [censored] encoding for another 2GB after it should have realised it was writing a broken file. Which in High Profile on a HD source is a [censored] lot of cycles. So even if I had been so bored as to watch the activity log while it was encoding I wouldn't have seen the error until it was too late anyway, would I? As it was, it was a batch running over several [censored] days and it would be [censored] useful if it had some kind of [censored] sensible default behaviour in case something [censored] happened, wouldn't it?

*sigh* Okay, listen, I don't think you've really examined what's going on here very closely.

HandBrake uses a library called mp4v2 to create MP4 files. Every time it encodes a sample of audio or video (like, a frame) it tells mp4v2 "Write this sample and get back to me": http://code.google.com/p/mp4v2/source/b ... 4track.cpp

And when it gets back to HB, HB checks to make sure it didn't return an error. If it did return an error, HB tells the user it cannot write to disk, suggests the possibility that maybe the disk is full, and immediately quits encoding: http://trac.handbrake.fr/browser/trunk/ ... p4.c#L1015

The error message you see in your logs, though, is *not* generated by HB. It's generated by mp4v2. If you search through their project you will see that the error is generated in a function called FinishWrite: http://code.google.com/p/mp4v2/source/b ... om.cpp#541

1. HB calls MP4WriteSampleDependency(): http://code.google.com/p/mp4v2/source/b ... ck.cpp#475

2. MP4WriteSampleDependency() calls MP4WriteSample(): http://code.google.com/p/mp4v2/source/b ... ck.cpp#401

3. MP4WriteSample() calls a number of functions to update atoms with information about the latest sample. These atom edits ultimately result in MP4AtomWrite(): http://code.google.com/p/mp4v2/source/b ... om.cpp#495

4. AtomWrite calls FinishWrite, which throws the assert once you're past 4GB, but AtomWrite doesn't check for errors.

So, given the current state of this third party code, how is HB supposed to know when that assert is thrown? It only results in a message being written to stderr, and the public function HB is calling explicitly tells it through its return state that an error has not occurred.

One question I suppose is why, if that's planned for 0.9.5, the presets on the current SVNs I'm running don't already have large file as the default

Because I haven't updated the presets yet. I said 0.9.5 would have it, I never said it was in the SVN now.

[BradleyS]

*gives everybody a hug*

[rachel]

*gives everybody a hug*

No, i've got the message. The message seems to be: don't make feature requests in the feature requests forum that our software design makes difficult to implement or you'll be treated like a moron. Feature requests forum is reserved for developers already familiar with the codebase.

[mduell]

In addition to being difficult to implement, it's stupid. It's like complaining you get mp4 files because you didn't switch the container to mkv. If you want large files, enable the [censored] checkbox.

[BradleyS]

Rejected hug. Ouch.

Seriously though, a few thoughts:

1) If you're exceeding 4GB, I hope it's on 1080p material. If you're exceeding this size with 720p or SD material, you're doing it wrong.

2) An intelligent switch between 32- and 64-bit (large) file output seems to make sense on the surface (and you're not wrong at all for suggesting it), but clearly doesn't make sense given that nearly all modern devices and players support 64-bit output. And as jbrjake explained, implementing it would require modification to the third-party muxer used in HB. It's far easier to just make 64-bit files the default...

3) It's going to be the default for 0.9.5. Unless you have a (much) older device that does not support it, it's safe to enable it for all encodes. There's no major issue with files < 4GB being 64-bit.

I encourage you not to take anyone's comments quite seriously. Especially mduell's. (Hey buddy. Yeah, I know, GFM.)

[jbrjake]

The message seems to be: don't make feature requests in the feature requests forum that our software design makes difficult to implement or you'll be treated like a moron. Feature requests forum is reserved for developers already familiar with the codebase.

Not at all. But when you get an admittedly "adequate" user-type response, and then start acting like a developer by making claims about the code that imply familiarity with the codebase:

Obviously it stops as soon as it finds the error

and:

But as it stands it's a fatal error that breaks the encode and demands user attention, and one line buried in the activity log is by no standards an adequate way of handling it. It even returns a success (0) result code! (last line of the log)

...you should expect to get treated like a moron when the things you are saying about the code simply are not accurate.

And by the way? This has absolutely nothing to do with our "software design" making things "difficult." I thought I made this clear: it's an issue in a third party library we use. I even pointed you to the very line in HB that is designed to do what you want--warn the user when the 3rd party muxer throws a fatal error and immediately stop encoding.

[dynaflash]

Um, you could always make your own presets with 64 bit mp4's checked until the built in presets get changed as long as all of your target devices support it. I hate to throw the onus on the user but .....

Like many things ... what seems simple on the surface is not always so. Its life. Live it. As has been said it will get changed in terms of the built in presets.

[jamiemlaw]

I realise this thread is old, but a couple of users have come to the forum recently experiencing this 'file will not play' issue.

So I understand that when the error is thrown, mp4v2 just ignores it without notifying HandBrake. Is there anything HandBrake can do to probe what has been written to stderr, and see if a line saying "MP4ERROR: (m_size <= (uint64_t)0xFFFFFFFF): assert failure" has been written to it?

Or even possibly check the file size once the encode has completed.

I guess the goal here is to end up with some way for the user to be alerted of this error beyond a cryptic message in the activity log.