Is there a lossless setting for merging 4GB Sony AVC chunks?

HandBrake for Windows support
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Sun Sep 25, 2016 9:10 am

Camera: Sony FDR-1000V
Format: 1080P 50FPS 25mbits/sec AVC
Codec: H264-MPEG-4 AVC (Part 10) (avc1)
1920x1080
Planar 4:2:0 YUV
Audio: MPEG AAC (mp4a) 48kHz (but audio can be stripped off)

The reason I need to merge the chunks into one big file is that the breaks break various features in the Sony Vegas video editor.

It isn't too hard to work out how to render them into one huge file but it takes about 10x real time so a 5hr input takes 50hrs to render. Whereas a lossless process should be much faster because no re-encoding is needed.

Very many thanks for any help.

I am running latest Handbrake under win7-64, i7-970, 24GB RAM, GTX750 2GB.

But I have another machine which can be used for processing: winXP SP3, Handbrake 0.9.9.5530 32 bit which AFAIK is the latest version to run under winXP. If this could be used, that would be great.

User avatar
s55
HandBrake Team
Posts: 9463
Joined: Sun Dec 24, 2006 1:05 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by s55 » Sun Sep 25, 2016 9:16 am

HandBrake cannot merge files, not does it support passthru so it is not the tool you are looking for.

peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Sun Sep 25, 2016 10:26 am

I thought HB could take in a list of files and output one file?

Or is the list of files mode just a means of batch processing multiple files, with each one generating a new output file?

Could anyone suggest another tool?

User avatar
s55
HandBrake Team
Posts: 9463
Joined: Sun Dec 24, 2006 1:05 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by s55 » Sun Sep 25, 2016 10:54 am

IT's batch processing, not file combining.

I don't typically combine files so can't really make any good recommendations.

musicvid
Veteran User
Posts: 3210
Joined: Sat Jun 27, 2009 1:19 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by musicvid » Sun Sep 25, 2016 11:43 am

On Windows, the tool is Videoredo

User avatar
Rodeo
HandBrake Team
Posts: 12121
Joined: Tue Mar 03, 2009 8:55 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Rodeo » Sun Sep 25, 2016 2:05 pm

Are all files 4GB (or are there some that are smaller), and how are they named?

peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Sun Sep 25, 2016 3:26 pm

They are all 4GB except the last one which is usually smaller.

The stupid camera (FDR-1000V) writes them like that even though the SD card is EXFAT...

The names are

MAH00665.MP4
MAH00666.MP4
MAH00667.MP4

I found various free file combiners (all involving adware/malware) which might work, and one non-free one - Machete (machetesoft.com) but haven't tested it yet.

Shane
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane » Sun Sep 25, 2016 4:22 pm

Pegasys Inc. has a Lossless MPEG editor called TMPGEnc MPEG Smart Renderer 5 that will do exactly what you want and only take a few minutes to do it because it doesn't re-render all the files. It only renders the few frames where the clips are put together and leaves the rest untouched.

You can use it completely unrestricted for 30 days. No watermarks, no time limits, no spyware. If you decide to keep it it's $70.

I use their file convertor/timeline editing software and Bluray/DVD authoring software and I really like their products. HB does a better job at h.264 than their convertor/timeline software but it outputs excellent Mpeg-2 files which I then throw in to HB.

They also have bundle deals.

Hope this helps.

Shane

User avatar
Rodeo
HandBrake Team
Posts: 12121
Joined: Tue Mar 03, 2009 8:55 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Rodeo » Sun Sep 25, 2016 5:13 pm

peterh337 wrote:They are all 4GB except the last one which is usually smaller.
:-(

So even if you record a 10-second clip, it will get appended to one of the large chunks instead of getting its own file?

musicvid
Veteran User
Posts: 3210
Joined: Sat Jun 27, 2009 1:19 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by musicvid » Sun Sep 25, 2016 7:08 pm

Those look like standard camera chunks on SD or p2. Camera is going to write 4GB no matter how the card is formatted.

Don't use any third partyj software; use the import utility that came with your camera, because it understands the splits, which are tricky with transport streams. Tmpgenc can get out of sync if there are lots of ts stream errors, I promise

peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Sun Sep 25, 2016 7:43 pm

The camera will write whatever size of file is required, with a 4GB limit on the size of the file.

This is required on FAT32 SD cards but in this case the camera does it anyway...

Apparently Sony do produce a program called Play Memories Home which can do this, but it is a massive piece of bloatware which tries to become your default picture and movie viewer, etc.

User avatar
Rodeo
HandBrake Team
Posts: 12121
Joined: Tue Mar 03, 2009 8:55 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Rodeo » Sun Sep 25, 2016 9:16 pm

One some camera models, they used to simply cut off the file at 4GB and keep on writing to the next without finalizing anything: you didn't have to merge them, just concatenate the files. No idea if it would work with this camera model, however (I'm guessing since it uses MP4 instead of transport streams, probably not).

peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Mon Sep 26, 2016 7:16 am

In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.

User avatar
Rodeo
HandBrake Team
Posts: 12121
Joined: Tue Mar 03, 2009 8:55 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Rodeo » Mon Sep 26, 2016 7:25 am

Assuming the video and audio track have the exact same length, it should be easy enough to combine them using avconv or ffmpeg.

kpflugshaupt
Posts: 63
Joined: Sat Feb 06, 2010 8:00 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by kpflugshaupt » Mon Sep 26, 2016 9:03 am

peterh337 wrote:In this case, no, each chunk is a totally standalone mp4 file. I can play each chunk with VLC, etc.

Otherwise one could join them with copy /b in a DOS box :)

The implementation is slightly interesting because even when the camera stops because the battery runs out, you get all valid chunks including the last one, so, obviously, the camera must be writing them in such a way that they are easy to finalise. In most movie format implementations you have an "atom" at the very end which the video player must retrieve to play the file at all. I would bet this camera doesn't do that and instead does what Handbrake has the "web friendly" checkbox for, where the atom is stored at the beginning.
It's also possible that the movie is getting "emergency-finished" when the device shuts itself down due to low charge. It would be the sensible thing to do, IMO. Rather shut down in a controlled way a bit too early than leave behind unfinished garbage!

Shane
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane » Tue Sep 27, 2016 4:16 pm

I'm not a Tmpgenc fanboi by no means. It's use of h.264 is really bad and it lacks a lot of authoring options but for the price, it's really hard to beat.

I've had it for a few months now and I've authored over 30 movies all with edits, subtitles, and multiple audio tracks using custom graphics and background videos, and thumbnail videos, created over 20 timeline intro videos about 3 minutes long each, several picture slide shows, combined several obscure tv shows that were split up on u-toob, converted dozens of video's and edited them and haven't lost sync once. I can't possibly imagine it would fail putting together a few video clips directly from the source but stranger things have happened I suppose. :)

It's worth a try since it's totally free and unrestricted for 30 days.

peterh337
Posts: 12
Joined: Sun Sep 25, 2016 8:23 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by peterh337 » Thu Sep 29, 2016 4:42 pm

I have now tested AviDemux and Machete. Both work great. AviDemux has a much better user interface than Machete, however, for merging files. With both programs one has to add each input file individually (there is no "import all" option) but with Machete (which is not free, despite the developer being totally unresponsive) it takes much longer because one has to slide the Play slider all the way to the right after adding each file, before adding the next one. The speed of both programs is good; should be since they don't re-encode.

User avatar
Rodeo
HandBrake Team
Posts: 12121
Joined: Tue Mar 03, 2009 8:55 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Rodeo » Sat Oct 01, 2016 3:01 pm

Glad you found a solution :-)

musicvid
Veteran User
Posts: 3210
Joined: Sat Jun 27, 2009 1:19 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by musicvid » Sat Oct 01, 2016 3:56 pm

Some cameras finish the trailing gop with a key frame, some do not, and some are a mystery. That's why your camera's dedicated software is your go-to choice.

PLAYABILITY IN "VLC, ETC." IS NO INDICATION WHATSOEVER OF WHETHER GOP TERMINATION IS CLEAN, DIRTY, OR NONEXISTENT, NOR OF THE PRESENCE OR ABSENCE OF ERRORED TS FRAMES !

By the same token, neither is playability of dos-concatenated mp4 files. We are only concerned with a seamless join and sustainable sync in an editor or encoder!

And of course, the placement of the moov atom does not affect any of this, only the load time.

Shane
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane » Sat Oct 01, 2016 7:18 pm

Wow! I had never heard of AviDemux. If I could somehow lump Handbrake's 264 capabilities with it I would be an extremely happy camper. I messed with it for hours this week and cannot get it to produce anywhere near as good a file as Handbrake does. Not saying it can't, just that I can't figure it out. It has my beloved custom w x h settings that you can set inside a 16:9 pad via gui and you can adjust the picture settings including sharpening them. Very nice program, but the output quality is greatly lacking.

musicvid
Veteran User
Posts: 3210
Joined: Sat Jun 27, 2009 1:19 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by musicvid » Sat Oct 01, 2016 10:01 pm

The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.

Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.

That said, Avidemux has no capabilities to correct or reindex errors that occur in TRANSPORT STREAMS, which SD/P2 footage consists of. It is not a free substitute for VRD or TSDoctor, which do all that and more. Nor will dropping muxed footage into an EDITOR or ENCODER, even wrapped as a PROGRAM STREAM cure any errors or TRUNCATED GOPs.

Interesting and useful vocab is in caps.
Last edited by musicvid on Sun Oct 02, 2016 2:31 am, edited 3 times in total.

musicvid
Veteran User
Posts: 3210
Joined: Sat Jun 27, 2009 1:19 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by musicvid » Sat Oct 01, 2016 10:19 pm

To the OP, use the software that came with the camera or the Device Explorer (NOT Windows Explorer!!) in Vegas Movie Studio Platinum. You really dont need or want to reinvent the wheel.....

Shane
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane » Sun Oct 02, 2016 4:33 pm

musicvid wrote:The differences beetween encoding and muxing are simply not known to most people, but i think we have been here before, recently.

MUXING creates a precise, exact bit-for-bit copy of the original. Perceived differences are physically impossible.
There are exceptions to that that I have run in to. There's been several rips that I wanted to simply remove the extra audio tracks from and remove the subtitle tracks from and the muxing software altered the resulting container file and even took it out of Bluray Compliance.
Avidemux is another utility that has been around since the beginning of time. It is a MUXER with some ENCODING capabilities, which are not very good with its INTERFRAME compressors.
I don't know when you checked it out last, but it has advanced x.264 encoding capabilities. Moreso than HB in a lot of ways, but the quality is not nearly as good as HB.

Not trying to be contradictory or anything like that, just saying that because it's in BOLD doesn't necessarily make it gospel. :wink:

mduell
Veteran User
Posts: 6548
Joined: Sat Apr 21, 2007 8:54 pm

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by mduell » Sun Oct 02, 2016 6:57 pm

Muxing makes no changes to the tracks themselves, but of course it can change the container.

Do you mean H.264 or x264? There is no x.264.

Shane
Posts: 57
Joined: Thu Aug 25, 2016 2:47 am

Re: Is there a lossless setting for merging 4GB Sony AVC chunks?

Post by Shane » Sun Oct 02, 2016 8:28 pm

I meant the default encoding that hb uses. I get confused sometimes with all these variations. Sorry about that.

But yeah, I've had several containers get messed up that I thought I was good to go with and it ended up setting me back days to fix.

For example this grayscale disc I've been working on for quite a while now that I was asking about in another thread. I'm still working on it. I'm getting frustrated because the manual is just terrible for hb. It assumes everyone knows everything about syntax and default settings and what settings are allowed. It doesn't seem like it should be a big deal to throw a quick per command sample. Sorry, getting off track. I've just been working all day again on a single dvd I'm trying to convert and it's not going well.

I'm very frustrated right now and should go.

Thanks.

Post Reply