Is there a lossless setting for merging 4GB Sony AVC chunks?

HandBrake for Windows support
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

Camera: Sony FDR-1000V
Format: 1080P 50FPS 25mbits/sec AVC
Codec: H264-MPEG-4 AVC (Part 10) (avc1)
1920x1080
Planar 4:2:0 YUV
Audio: MPEG AAC (mp4a) 48kHz (but audio can be stripped off)

The reason I need to merge the chunks into one big file is that the breaks break various features in the Sony Vegas video editor.

It isn't too hard to work out how to render them into one huge file but it takes about 10x real time so a 5hr input takes 50hrs to render. Whereas a lossless process should be much faster because no re-encoding is needed.

Very many thanks for any help.

I am running latest Handbrake under win7-64, i7-970, 24GB RAM, GTX750 2GB.

But I have another machine which can be used for processing: winXP SP3, Handbrake 0.9.9.5530 32 bit which AFAIK is the latest version to run under winXP. If this could be used, that would be great.
User avatar
s55
HandBrake Team
Posts: 10350
Joined: Sun Dec 24, 2006 1:05 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by s55 »

HandBrake cannot merge files, not does it support passthru so it is not the tool you are looking for.
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

I thought HB could take in a list of files and output one file?

Or is the list of files mode just a means of batch processing multiple files, with each one generating a new output file?

Could anyone suggest another tool?
User avatar
s55
HandBrake Team
Posts: 10350
Joined: Sun Dec 24, 2006 1:05 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by s55 »

IT's batch processing, not file combining.

I don't typically combine files so can't really make any good recommendations.
Deleted User 13735

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 13735 »

On Windows, the tool is Videoredo
Deleted User 11865

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 11865 »

Are all files 4GB (or are there some that are smaller), and how are they named?
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

They are all 4GB except the last one which is usually smaller.

The stupid camera (FDR-1000V) writes them like that even though the SD card is EXFAT...

The names are

MAH00665.MP4
MAH00666.MP4
MAH00667.MP4

I found various free file combiners (all involving adware/malware) which might work, and one non-free one - Machete (machetesoft.com) but haven't tested it yet.
Shane
Novice
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane »

Pegasys Inc. has a Lossless MPEG editor called TMPGEnc MPEG Smart Renderer 5 that will do exactly what you want and only take a few minutes to do it because it doesn't re-render all the files. It only renders the few frames where the clips are put together and leaves the rest untouched.

You can use it completely unrestricted for 30 days. No watermarks, no time limits, no spyware. If you decide to keep it it's $70.

I use their file convertor/timeline editing software and Bluray/DVD authoring software and I really like their products. HB does a better job at h.264 than their convertor/timeline software but it outputs excellent Mpeg-2 files which I then throw in to HB.

They also have bundle deals.

Hope this helps.

Shane
Deleted User 11865

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 11865 »

peterh337 wrote:They are all 4GB except the last one which is usually smaller.
:-(

So even if you record a 10-second clip, it will get appended to one of the large chunks instead of getting its own file?
Deleted User 13735

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 13735 »

Those look like standard camera chunks on SD or p2. Camera is going to write 4GB no matter how the card is formatted.

Don't use any third partyj software; use the import utility that came with your camera, because it understands the splits, which are tricky with transport streams. Tmpgenc can get out of sync if there are lots of ts stream errors, I promise
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

The camera will write whatever size of file is required, with a 4GB limit on the size of the file.

This is required on FAT32 SD cards but in this case the camera does it anyway...

Apparently Sony do produce a program called Play Memories Home which can do this, but it is a massive piece of bloatware which tries to become your default picture and movie viewer, etc.
Deleted User 11865

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 11865 »

One some camera models, they used to simply cut off the file at 4GB and keep on writing to the next without finalizing anything: you didn't have to merge them, just concatenate the files. No idea if it would work with this camera model, however (I'm guessing since it uses MP4 instead of transport streams, probably not).
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.
Deleted User 11865

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 11865 »

Assuming the video and audio track have the exact same length, it should be easy enough to combine them using avconv or ffmpeg.
kpflugshaupt
Novice
Posts: 63
Joined: Sat Feb 06, 2010 8:00 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by kpflugshaupt »

peterh337 wrote:In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.
It's also possible that the movie is getting "emergency-finished" when the device shuts itself down due to low charge. It would be the sensible thing to do, IMO. Rather shut down in a controlled way a bit too early than leave behind unfinished garbage!
Shane
Novice
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane »

I'm not a Tmpgenc fanboi by no means. It's use of h.264 is really bad and it lacks a lot of authoring options but for the price, it's really hard to beat.

I've had it for a few months now and I've authored over 30 movies all with edits, subtitles, and multiple audio tracks using custom graphics and background videos, and thumbnail videos, created over 20 timeline intro videos about 3 minutes long each, several picture slide shows, combined several obscure tv shows that were split up on u-toob, converted dozens of video's and edited them and haven't lost sync once. I can't possibly imagine it would fail putting together a few video clips directly from the source but stranger things have happened I suppose. :)

It's worth a try since it's totally free and unrestricted for 30 days.
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 »

I have now tested AviDemux and Machete. Both work great. AviDemux has a much better user interface than Machete, however, for merging files. With both programs one has to add each input file individually (there is no "import all" option) but with Machete (which is not free, despite the developer being totally unresponsive) it takes much longer because one has to slide the Play slider all the way to the right after adding each file, before adding the next one. The speed of both programs is good; should be since they don't re-encode.
Deleted User 11865

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 11865 »

Glad you found a solution :-)
Deleted User 13735

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 13735 »

Some cameras finish the trailing gop with a key frame, some do not, and some are a mystery. That's why your camera's dedicated software is your go-to choice.

PLAYABILITY IN "VLC, ETC." IS NO INDICATION WHATSOEVER OF WHETHER GOP TERMINATION IS CLEAN, DIRTY, OR NONEXISTENT, NOR OF THE PRESENCE OR ABSENCE OF ERRORED TS FRAMES !

By the same token, neither is playability of dos-concatenated mp4 files. We are only concerned with a seamless join and sustainable sync in an editor or encoder!

And of course, the placement of the moov atom does not affect any of this, only the load time.
Shane
Novice
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane »

Wow! I had never heard of AviDemux. If I could somehow lump Handbrake's 264 capabilities with it I would be an extremely happy camper. I messed with it for hours this week and cannot get it to produce anywhere near as good a file as Handbrake does. Not saying it can't, just that I can't figure it out. It has my beloved custom w x h settings that you can set inside a 16:9 pad via gui and you can adjust the picture settings including sharpening them. Very nice program, but the output quality is greatly lacking.
Deleted User 13735

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 13735 »

The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.

Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.

That said, Avidemux has no capabilities to correct or reindex errors that occur in TRANSPORT STREAMS, which SD/P2 footage consists of. It is not a free substitute for VRD or TSDoctor, which do all that and more. Nor will dropping muxed footage into an EDITOR or ENCODER, even wrapped as a PROGRAM STREAM cure any errors or TRUNCATED GOPs.

Interesting and useful vocab is in caps.
Last edited by Deleted User 13735 on Sun Oct 02, 2016 2:31 am, edited 3 times in total.
Deleted User 13735

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Deleted User 13735 »

To the OP, use the software that came with the camera or the Device Explorer (NOT Windows Explorer!!) in Vegas Movie Studio Platinum. You really dont need or want to reinvent the wheel.....
Shane
Novice
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane »

musicvid wrote:The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.
There are exceptions to that that I have run in to. There's been several rips that I wanted to simply remove the extra audio tracks from and remove the subtitle tracks from and the muxing software altered the resulting container file and even took it out of Bluray Compliance.
Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.
I don't know when you checked it out last, but it has advanced x.264 encoding capabilities. Moreso than HB in a lot of ways, but the quality is not nearly as good as HB.

Not trying to be contradictory or anything like that, just saying that because it's in BOLD doesn't necessarily make it gospel. :wink:
mduell
Veteran User
Posts: 8187
Joined: Sat Apr 21, 2007 8:54 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by mduell »

Muxing makes no changes to the tracks themselves, but of course it can change the container.

Do you mean H.264 or x264? There is no x.264.
Shane
Novice
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane »

I meant the default encoding that hb uses. I get confused sometimes with all these variations. Sorry about that.

But yeah, I've had several containers get messed up that I thought I was good to go with and it ended up setting me back days to fix.

For example this grayscale disc I've been working on for quite a while now that I was asking about in another thread. I'm still working on it. I'm getting frustrated because the manual is just terrible for hb. It assumes everyone knows everything about syntax and default settings and what settings are allowed. It doesn't seem like it should be a big deal to throw a quick per command sample. Sorry, getting off track. I've just been working all day again on a single dvd I'm trying to convert and it's not going well.

I'm very frustrated right now and should go.

Thanks.
Post Reply