trouble detelecining 720p hdtv captures

HandBrake for Windows support
Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
Post Reply
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

trouble detelecining 720p hdtv captures

Post by rcrh »

I'm working with the SVN2845 Snapshot and am having problems dropping repeated frames.

My source files are 720p hdtv captures broadcast at 59.97fps.

I thing the source material is film at 23.976fps because when I open the files in mediaplayer classic and step through them frame by frame I can see (n=new frame, r=repeat) NRNRRNRNRR...

I've found that I get very good results if I force the fps down to 23.976 and use the decomb filter. But after every commercial break it seems to be a crapshoot as to whether or not handbrake is dropping the right frames. After some commercial cuts I get nothing but new frames but after other cuts I get NNNRNNNR (every fourth frame is a repeat and I suspect a new frame has been dropped as well). The resulting file plays smoothly in some sections and is choppy in others.

I've tried the detelecine filter but I need up with files that are still 59.97fps.

Is there anything I should be doing to get handbrake to sync up the frame count after commercial breaks or is the problem back in the software that I'm using to remove the commercials?

Thanks in advance for any suggestions.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: trouble detelecining 720p hdtv captures

Post by jbrjake »

How is the source telecined? Are the telecine frames combed (lines alternate vertically between two moments in time), or blended (where it looks like two moments in time blended together) ?

And really, a log is *required*. i cannot begin to help you without one.

A very short sample of the source (<30 sec) of a problem area would be helpful, too.
tlindgren
Bright Spark User
Posts: 243
Joined: Sun May 03, 2009 2:14 pm

Re: trouble detelecining 720p hdtv captures

Post by tlindgren »

jbrjake wrote:How is the source telecined? Are the telecine frames combed (lines alternate vertically between two moments in time), or blended (where it looks like two moments in time blended together) ?
From the description it sounds like 24 fps progressive video stepped up to 60 fps with frame duplication, he mentions identical frames without combing or blending. If he's unlucky they've blended in 60i content into it like animated or moving station logo, subtitles or side-scrolling text :!:

I'm not sure this is "telecine" as such but it's not uncommon to call it that and many other deteleciner's can handle it, though usually they work best if they're told told the target fps or decimate ratio because it can be very hard to detect "identical" frames (due to noise or other factors), then it needs to either select one or blend together the most similar ones (free denoising, when it works!). But you've earlier indicated that setting output fps is a bad idea for HandBrake and I trust you know better (I didn't see the output fps in the detelecine code so it makes sense).

I had some really nasty sources like that earlier, the sections with changing logo messed up all standard deteleciners I tried though most handled most of the video with proper settings (unfortunately there were pans where the logo changed too in 60i!, kabloie). With Handbrake (semi-recent) with standard detelecine/decomb I got the same number of frames in and out according to the log, I think that means it didn't drop any frames (due to the noise making it think all frames were different?).

I did manage to get perfect 24 fps result from AviSynth (slightly smoother than the 60 fps input!) but it was very custom, I had to use a deteleciner which allowed me to use a different video with logo area removed for frame selection but then used the real video for output. For some reason all other fps reductions I tried was really interested in that ***** logo...
jbrjake wrote:A very short sample of the source (<30 sec) of a problem area would be helpful, too.
Definitely, it will confirm what kind of video stream it is and what can be done. The log (without forced fps) will also give a lot of details on what is going on as you note.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Ok first, sorry for not posted the required files along with the request. I had quickly scanned the log and didn't see any relevant errors or warnings. But, here it is now.

http://handbrake.fr/pastebin/pastebin.php?show=950

Second, I've created a 19 second clip of the source file but I'm not sure how to make it available.

Third, I've got a bigger problem in that I made the source clip around a section that shows the problem in the full encode (frames are all new before the scene change but are nnnrnnnr after) but when I encode this it turns out fine. All frames are new.

Truly weird.

Finally, I think I was wrong in the subject line with I described the problem as detelecining. I'm not sure that the source is telecined. It's likely 24 frames that have been stepped up to 60 frames.

If someone can help me with submitted the sample I'd be glad to provide that too.

Thanks again. I really appreciate the help.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: trouble detelecining 720p hdtv captures

Post by jbrjake »

A log of an encode where you used detelecine would be much better.

This is nonsense:

Code: Select all

--decomb="pullup" 
Detelecine is enabled with --detelecine and --decomb does not take parameters in the form of English words.

You can put a clip on a site like drop.io or mediafire.com, and post a link. A clip that still shows the problem after editing would be a lot more helpful, if you can manage to find one...I know it's difficult, everything changes when you edit.

@tlindgren: This is a Fox show produced at 24fps and broadcast at 720p. It's the cable company pushing it up to 59.9 to run over 1080i streams. This often is done by blending the telecine frames (for example, that's how dynaflash's TV provider sends him Lost, another 720/24p series). A number of times in the past, people have said the frames were duplicates just because they didn't look closely enough to see the blending (it can be difficult, especially if you look at a relatively still portion and have the resolution scaled down so you can see the whole thing on a < 1080p display). So I like to verify. As with Fringe, when it comes to telecining, there's more than one of everything.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

more lies on my part! or maybe operator errors.

I didn't include the detelecine log because the resulting file ended up with 60fps and all of the repeated frames. So, I concluded that the file was telecined. The decomb filter seemed to mostly work.

here is the sample source file

http://drop.io/hb817817817

in a related question, what are the options for the detelecine & decomb filters? I thought I had found something be I guess I got it wrong.

thanks again.
tlindgren
Bright Spark User
Posts: 243
Joined: Sun May 03, 2009 2:14 pm

Re: trouble detelecining 720p hdtv captures

Post by tlindgren »

rcrh wrote:I didn't include the detelecine log because the resulting file ended up with 60fps and all of the repeated frames. So, I concluded that the file was telecined. The decomb filter seemed to mostly work.
here is the sample source file: http://drop.io/hb817817817
The file looks like I expected, it appears to be 24p converted to 60p with frame duplication but due to noise (presumably from the MPEG2 encoding/decoding) every single frame is "different". There's no combing or blending in areas where it ought to show if it was there, and the full log you showed earlier shows that decomb only processed 2 of 175437 frames on the full source so HandBrake clearly agreed.

I ran your sample through my recent SVN checkout and like you I get the same number of frames coming in and out (with decomb+detelecine at default settings). I played a bit with custom detelecine settings without luck, I'm not ruling out that it can be done but I don't see any obvious settings for this.

So I feed it into AviSynth and used Donald A. Graft's standard Decomb plugin, using Decimate(cycle=2) followed by Decimate(cycle=5), to generate 23.976 fps output, the result looks perfectly smooth. Someone else has a SmartDeinterlace that can do any reduction in one step (and ignore things like animated logos) but I've found it's frame selection much worse which can lead to stutter especially around cadence breaks, these can happen due to things like editing or commercials.

I'm fairly confident that it would give good results on the full file but obviously I can't prove that. On the other hand it requires a bit of work and not using HandBrake (unfortunately it doesn't understand AviSynth files so you need another x264 encoder). You have to decide if it's worth it considering that the output is fine in 60 fps. Hmm, doing some sample encodes Q62 suggests the size drawback for this sample is considerable, much larger than I expected to be frank. Granted, it's possible that the quality metric has problem with this or that it's different for the full file but I didn't expect the 60 fps file to be almost 70-80% larger! It does have 2.5 times the number of frames so the bytes per frame is actually lower but it's still unfortunate.

I tried fairly heavy temporal and spatial noise reduction before feeding the result to HandBrake and only managed to get it to drop 6 frames, on the input the frames [do b]look[/b] identical but apparently they need to be absolutely identical which I doubt is going to happen from a HDTV stream without far heavier denoising than can be done without loosing a lot of the quality! But it does show HB will drop identical frames to generate VFR output.

However if jbrjake can show how to get HandBrake to convert this to 24 fps via detelecine it would be great for me too, it would certainly make several things a lot simpler.
rcrh wrote:in a related question, what are the options for the detelecine & decomb filters? I thought I had found something be I guess I got it wrong.
All of them accept a string containing numbers separated with colons, you need to get it exactly correct or it won't take effect. On Windows "HandBrakeCLI.exe --help" will show you a bit of information but it looks like most or all of the "defaults" it lists are incorrect in recent SVN versions & snapshots.
To find the actual default you can consult the source (libhb/decomb.c, libhb/deinterlace.c & libhb/detelecine.c), though it requires some knowledge and with the exception of decomb there's very little information on what the values does. Decomb documents the first parameter pretty good but the others are less well documented, none of them are for the faint of heart.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Well, even though tlindgren's post makes it look like I'm hitting a dead end with handbrake I did find another example of the same problem and this is one where I get the behaviour with both the full file & the sample.

So, I'm now linking to:

the sample file
http://drop.io/hb817817817_2
the log file
http://drop.io/hb817817817_3
the encoded file
http://drop.io/hb817817817_4

I hope this someone takes another kick at this and might come up with something.

thanks again.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

It looks like I'm going to have to give up on Handbrake for now.

I tried the same files with another front end and the encodes came out fine.

Too bad 'cause I liked working with HB.
jbrjake
Veteran User
Posts: 4805
Joined: Wed Dec 13, 2006 1:38 am

Re: trouble detelecining 720p hdtv captures

Post by jbrjake »

Sorry, haven't had a chance to check out the sample yet, although it's been added to my test sample directory.

No idea what you mean about "another front end"...HB isn't a frontend to another program. Whatever you used functions differently, not just presents it differently or using different options to the same underlying code. Would need to know, for example, which app you used and what filters and settings you used in it.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Sorry it's been so long seems I've been back here.

What I meant by "another from end" is a front end to the x264 encoder.

I've been having success encoding these files using ripbot264 and a filter it uses called "decimate". It's worked with everyone of these files & the others that I get that are 1080i and 59 frames.

If you do get a chance I'd still love to be able to move back to HD.

thanks.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Wow, I can't believe I'm bumping a thread that's over two years old! But...

I see that nobody had any luck with second samples I put up. I'm still struggling with my quest to find a good solution for these OTA HDTV captures. I had been using ripbot264 but then noticed that sometimes it would give me jerky results and I went back to playing with handbrake. Now my best results seem to be with HB and forcing the framecount to 29.97. I still end up with duplicate frames but nothing like when I try to produce 23.976 fps.

I'm linking to another sample file. This zip contains a three minute source file, a HB encode at 29.97fps, an HB encode at 23.976fps, and a ripbot encode at 23.976. The 29fps encode is the only one that plays smoothly and I'm happy with that except for the 20% increase in file size.

http://dl.dropbox.com/u/16279671/loci.s ... ive%29.zip

If folks could have a look and make any suggestions I would be greatly appreciative.

Thanks in advance.
User avatar
JohnAStebbins
HandBrake Team
Posts: 5583
Joined: Sat Feb 09, 2008 7:21 pm

Re: trouble detelecining 720p hdtv captures

Post by JohnAStebbins »

I haven't tried your samples, but what version of HandBrake are you using now? There were changes made a short while ago that may help you if you are not using a recent nightly build already.

Commit
https://trac.handbrake.fr/changeset/4297

And here's the thread where this topic was discussed at length
viewtopic.php?f=13&t=22220&hilit=59+94
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

John, I'm on the official release. I'll have a look at the thread you've referenced and will try the nightly if for no other reason then to have another reference point.

I do you hope you get a chance to look at my samples though.

Thanks for the help. I'll report back once I look at the nightly.

Richard
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

John, great result on my three minute test file. I've now queued up a couple of full episodes and I'll encode over night with the SVN. I'll report back once I have a chance to review the result but I'm very hopeful at the moment.

Thanks for the help!
Richard
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

I just watched the first encode of one of these 720p 59fps files and it looks very good. There are a couple of places with handbrake has duplicated a frame (and I'm assuming dropped a good frame) but it's hardly noticeable. I can certainly live with this result and I greatly appreciate the approximately 20% drop in file size!

John, thanks again for pointing me to the thread discussing the SVN.

I'll report back if I notice a change in my results as I watch additional files.

Richard.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Well, my revised opinion after watching a few more episodes is that while I'm happier with the result it really isn't great. I'm still ending up with quite a few duplicate frames (and therefore lost frames too). I hate the idea of going back to 29.97fps but the herky-jerky motion is disappointing.

I'm hoping someone else picks this up and has another suggestion.
User avatar
JohnAStebbins
HandBrake Team
Posts: 5583
Joined: Sat Feb 09, 2008 7:21 pm

Re: trouble detelecining 720p hdtv captures

Post by JohnAStebbins »

One suggestion would be to create and supply a short clip that I can use to reproduce the problem and then maybe I can improve our algorithm some more.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

JohnAStebbins wrote:One suggestion would be to create and supply a short clip that I can use to reproduce the problem and then maybe I can improve our algorithm some more.
I did, but here it is again.

http://dl.dropbox.com/u/16279671/loci.s ... ive%29.zip

Hopefully you'll reproduce the same stutter.

Thanks.
User avatar
JohnAStebbins
HandBrake Team
Posts: 5583
Joined: Sat Feb 09, 2008 7:21 pm

Re: trouble detelecining 720p hdtv captures

Post by JohnAStebbins »

oops. I should have re-read the thread. thanks for the clip.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

JohnAStebbins wrote:oops. I should have re-read the thread. thanks for the clip.
No worries. I hope it works out (selfish jerk that I am). :D
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

JohnAStebbins wrote:oops. I should have re-read the thread. thanks for the clip.
John, did you get a chance to look at those files? Were you able to reproduce and then hopefully eliminate the stutter?

Thanks,
Richard
User avatar
JohnAStebbins
HandBrake Team
Posts: 5583
Joined: Sat Feb 09, 2008 7:21 pm

Re: trouble detelecining 720p hdtv captures

Post by JohnAStebbins »

No, I've been traveling and will be away for another week. So I'll have some time to look at this in a little over a week.
rcrh
Posts: 40
Joined: Wed Feb 11, 2009 1:49 am

Re: trouble detelecining 720p hdtv captures

Post by rcrh »

Much appreciated John. I'll sit tight. Safe travels.

Richard
Post Reply