Page 1 of 2

Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 9:10 am
by peterh337
Camera: Sony FDR-1000V
Format: 1080P 50FPS 25mbits/sec AVC
Codec: H264-MPEG-4 AVC (Part 10) (avc1)
1920x1080
Planar 4:2:0 YUV
Audio: MPEG AAC (mp4a) 48kHz (but audio can be stripped off)

The reason I need to merge the chunks into one big file is that the breaks break various features in the Sony Vegas video editor.

It isn't too hard to work out how to render them into one huge file but it takes about 10x real time so a 5hr input takes 50hrs to render. Whereas a lossless process should be much faster because no re-encoding is needed.

Very many thanks for any help.

I am running latest Handbrake under win7-64, i7-970, 24GB RAM, GTX750 2GB.

But I have another machine which can be used for processing: winXP SP3, Handbrake 0.9.9.5530 32 bit which AFAIK is the latest version to run under winXP. If this could be used, that would be great.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 9:16 am
by s55
HandBrake cannot merge files, not does it support passthru so it is not the tool you are looking for.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 10:26 am
by peterh337
I thought HB could take in a list of files and output one file?

Or is the list of files mode just a means of batch processing multiple files, with each one generating a new output file?

Could anyone suggest another tool?

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 10:54 am
by s55
IT's batch processing, not file combining.

I don't typically combine files so can't really make any good recommendations.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 11:43 am
by musicvid
On Windows, the tool is Videoredo

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 2:05 pm
by Rodeo
Are all files 4GB (or are there some that are smaller), and how are they named?

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 3:26 pm
by peterh337
They are all 4GB except the last one which is usually smaller.

The stupid camera (FDR-1000V) writes them like that even though the SD card is EXFAT...

The names are

MAH00665.MP4
MAH00666.MP4
MAH00667.MP4

I found various free file combiners (all involving adware/malware) which might work, and one non-free one - Machete (machetesoft.com) but haven't tested it yet.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 4:22 pm
by Shane
Pegasys Inc. has a Lossless MPEG editor called TMPGEnc MPEG Smart Renderer 5 that will do exactly what you want and only take a few minutes to do it because it doesn't re-render all the files. It only renders the few frames where the clips are put together and leaves the rest untouched.

You can use it completely unrestricted for 30 days. No watermarks, no time limits, no spyware. If you decide to keep it it's $70.

I use their file convertor/timeline editing software and Bluray/DVD authoring software and I really like their products. HB does a better job at h.264 than their convertor/timeline software but it outputs excellent Mpeg-2 files which I then throw in to HB.

They also have bundle deals.

Hope this helps.

Shane

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 5:13 pm
by Rodeo
peterh337 wrote:They are all 4GB except the last one which is usually smaller.
:-(

So even if you record a 10-second clip, it will get appended to one of the large chunks instead of getting its own file?

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 7:08 pm
by musicvid
Those look like standard camera chunks on SD or p2. Camera is going to write 4GB no matter how the card is formatted.

Don't use any third partyj software; use the import utility that came with your camera, because it understands the splits, which are tricky with transport streams. Tmpgenc can get out of sync if there are lots of ts stream errors, I promise

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 7:43 pm
by peterh337
The camera will write whatever size of file is required, with a 4GB limit on the size of the file.

This is required on FAT32 SD cards but in this case the camera does it anyway...

Apparently Sony do produce a program called Play Memories Home which can do this, but it is a massive piece of bloatware which tries to become your default picture and movie viewer, etc.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Sep 25, 2016 9:16 pm
by Rodeo
One some camera models, they used to simply cut off the file at 4GB and keep on writing to the next without finalizing anything: you didn't have to merge them, just concatenate the files. No idea if it would work with this camera model, however (I'm guessing since it uses MP4 instead of transport streams, probably not).

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Mon Sep 26, 2016 7:16 am
by peterh337
In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Mon Sep 26, 2016 7:25 am
by Rodeo
Assuming the video and audio track have the exact same length, it should be easy enough to combine them using avconv or ffmpeg.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Mon Sep 26, 2016 9:03 am
by kpflugshaupt
peterh337 wrote:In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.
It's also possible that the movie is getting "emergency-finished" when the device shuts itself down due to low charge. It would be the sensible thing to do, IMO. Rather shut down in a controlled way a bit too early than leave behind unfinished garbage!

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Tue Sep 27, 2016 4:16 pm
by Shane
I'm not a Tmpgenc fanboi by no means. It's use of h.264 is really bad and it lacks a lot of authoring options but for the price, it's really hard to beat.

I've had it for a few months now and I've authored over 30 movies all with edits, subtitles, and multiple audio tracks using custom graphics and background videos, and thumbnail videos, created over 20 timeline intro videos about 3 minutes long each, several picture slide shows, combined several obscure tv shows that were split up on u-toob, converted dozens of video's and edited them and haven't lost sync once. I can't possibly imagine it would fail putting together a few video clips directly from the source but stranger things have happened I suppose. :)

It's worth a try since it's totally free and unrestricted for 30 days.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Thu Sep 29, 2016 4:42 pm
by peterh337
I have now tested AviDemux and Machete. Both work great. AviDemux has a much better user interface than Machete, however, for merging files. With both programs one has to add each input file individually (there is no "import all" option) but with Machete (which is not free, despite the developer being totally unresponsive) it takes much longer because one has to slide the Play slider all the way to the right after adding each file, before adding the next one. The speed of both programs is good; should be since they don't re-encode.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sat Oct 01, 2016 3:01 pm
by Rodeo
Glad you found a solution :-)

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sat Oct 01, 2016 3:56 pm
by musicvid
Some cameras finish the trailing gop with a key frame, some do not, and some are a mystery. That's why your camera's dedicated software is your go-to choice.

PLAYABILITY IN "VLC, ETC." IS NO INDICATION WHATSOEVER OF WHETHER GOP TERMINATION IS CLEAN, DIRTY, OR NONEXISTENT, NOR OF THE PRESENCE OR ABSENCE OF ERRORED TS FRAMES !

By the same token, neither is playability of dos-concatenated mp4 files. We are only concerned with a seamless join and sustainable sync in an editor or encoder!

And of course, the placement of the moov atom does not affect any of this, only the load time.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sat Oct 01, 2016 7:18 pm
by Shane
Wow! I had never heard of AviDemux. If I could somehow lump Handbrake's 264 capabilities with it I would be an extremely happy camper. I messed with it for hours this week and cannot get it to produce anywhere near as good a file as Handbrake does. Not saying it can't, just that I can't figure it out. It has my beloved custom w x h settings that you can set inside a 16:9 pad via gui and you can adjust the picture settings including sharpening them. Very nice program, but the output quality is greatly lacking.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sat Oct 01, 2016 10:01 pm
by musicvid
The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.

Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.

That said, Avidemux has no capabilities to correct or reindex errors that occur in TRANSPORT STREAMS, which SD/P2 footage consists of. It is not a free substitute for VRD or TSDoctor, which do all that and more. Nor will dropping muxed footage into an EDITOR or ENCODER, even wrapped as a PROGRAM STREAM cure any errors or TRUNCATED GOPs.

Interesting and useful vocab is in caps.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sat Oct 01, 2016 10:19 pm
by musicvid
To the OP, use the software that came with the camera or the Device Explorer (NOT Windows Explorer!!) in Vegas Movie Studio Platinum. You really dont need or want to reinvent the wheel.....

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Oct 02, 2016 4:33 pm
by Shane
musicvid wrote:The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.
There are exceptions to that that I have run in to. There's been several rips that I wanted to simply remove the extra audio tracks from and remove the subtitle tracks from and the muxing software altered the resulting container file and even took it out of Bluray Compliance.
Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.
I don't know when you checked it out last, but it has advanced x.264 encoding capabilities. Moreso than HB in a lot of ways, but the quality is not nearly as good as HB.

Not trying to be contradictory or anything like that, just saying that because it's in BOLD doesn't necessarily make it gospel. :wink:

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Oct 02, 2016 6:57 pm
by mduell
Muxing makes no changes to the tracks themselves, but of course it can change the container.

Do you mean H.264 or x264? There is no x.264.

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Posted: Sun Oct 02, 2016 8:28 pm
by Shane
I meant the default encoding that hb uses. I get confused sometimes with all these variations. Sorry about that.

But yeah, I've had several containers get messed up that I thought I was good to go with and it ended up setting me back days to fix.

For example this grayscale disc I've been working on for quite a while now that I was asking about in another thread. I'm still working on it. I'm getting frustrated because the manual is just terrible for hb. It assumes everyone knows everything about syntax and default settings and what settings are allowed. It doesn't seem like it should be a big deal to throw a quick per command sample. Sorry, getting off track. I've just been working all day again on a single dvd I'm trying to convert and it's not going well.

I'm very frustrated right now and should go.

Thanks.