Combing Detection
Posted: Fri Feb 08, 2008 10:53 pm
The Transcode project has a filter called 32detect. It's for identifying telecined and/or interlaced content:
http://cvs.exit1.org/cgi-bin/viewcvs.cg ... &view=auto
It's got a simple little comb detector called interlace_test(). For each frame it's passed, it examines it in groups of 4 horizontal lines. If the first line's color is sufficiently close to the third line's color, and is sufficiently different from the second line's color, it marks it down as a potential interlacing artifact in the cc_1 metric. It does the same for line 2 versus line 4 and line 3 in the cc_2 metric. Then, it moves on to the next group of 4 lines, and repeats the process for the whole frame buffer. A final metric is calculated by adding together cc_1 and cc_2, multiplying by 1000, and dividing by the number of squared pixels in the frame. If this metric is high enough, the frame is flagged as interlaced.
I've shoehorned the code into libhb/deinterlace.c as detect_comb(), and arranged it so the actual deinterlacing functions are only called when the 32detect code senses combing. Erring on the side of caution, I've set the threshold *very* low -- 0, whereas transcode uses 9. This is because I want everything deinterlaced except perfect progressive frames. If there's even a hint of similarity between alternating lines, I'd rather play on the safe side and deinterlace it. These thresholds still need to be played with -- there's probably something to be said for lowering the difference threshold between alternating lines and increasing the equality threshold for sequential lines.
It's working great with fast (ffmpeg) deinterlacing. I haven't quite smoothed out the kinks with the motion deinterlacers. The problem is yadif stores a frame ahead...I spent all day fooling with different ways of setting it up, but so far the best I've done is sucky: tell it to output the input frame when there's no combing, but use the cached frame where there is combing. This causes occasional out-of-order frames. Just a proof of concept, for now ;> I'd appreciate any help anyone could give me on this part.
When everything's working right with yadif, this will means it'll be possible to run deinterlacing without taking forever, or messing up progressive frames (as is yadif's wont, sadly)... because it never actually gets called on them. Same for hard telecine stuff that's been successfully IVTC'd. But it will catch things IVTC misses, as well as interlaced parts of hybrid content. It seems to take up around 5% of HandBrake's work loop load when it's running on progressive content, compared to 10% for vfr and 10% for scaling.
So, with pullup running first, followed by this comb detection and then one of the deinterlacers, I think we're approaching a one-click "set it and forget it" way of handling any NTSC content quirks. Of course, we could just run yadif all the time without checking if it's needed, but that's a lot slower and will lose detail on progressive frames. Plus, it should even save time on fully interlaced material, by skipping the deinterlacer during static shots. If we want to go for something grander, it seems to me that we could use a delay queue (expand the one used by yadif) and compare lines across frames.
http://pastebin.ca/896753
Oh...if you're running this at the same time as VFR, be aware that you need to include --detelecine in your command line as well as --vfr. Otherwise, the deinterlacing filter will be applied first. Small issue I have to correct with VFR and the filter chain order.
http://cvs.exit1.org/cgi-bin/viewcvs.cg ... &view=auto
It's got a simple little comb detector called interlace_test(). For each frame it's passed, it examines it in groups of 4 horizontal lines. If the first line's color is sufficiently close to the third line's color, and is sufficiently different from the second line's color, it marks it down as a potential interlacing artifact in the cc_1 metric. It does the same for line 2 versus line 4 and line 3 in the cc_2 metric. Then, it moves on to the next group of 4 lines, and repeats the process for the whole frame buffer. A final metric is calculated by adding together cc_1 and cc_2, multiplying by 1000, and dividing by the number of squared pixels in the frame. If this metric is high enough, the frame is flagged as interlaced.
I've shoehorned the code into libhb/deinterlace.c as detect_comb(), and arranged it so the actual deinterlacing functions are only called when the 32detect code senses combing. Erring on the side of caution, I've set the threshold *very* low -- 0, whereas transcode uses 9. This is because I want everything deinterlaced except perfect progressive frames. If there's even a hint of similarity between alternating lines, I'd rather play on the safe side and deinterlace it. These thresholds still need to be played with -- there's probably something to be said for lowering the difference threshold between alternating lines and increasing the equality threshold for sequential lines.
It's working great with fast (ffmpeg) deinterlacing. I haven't quite smoothed out the kinks with the motion deinterlacers. The problem is yadif stores a frame ahead...I spent all day fooling with different ways of setting it up, but so far the best I've done is sucky: tell it to output the input frame when there's no combing, but use the cached frame where there is combing. This causes occasional out-of-order frames. Just a proof of concept, for now ;> I'd appreciate any help anyone could give me on this part.
When everything's working right with yadif, this will means it'll be possible to run deinterlacing without taking forever, or messing up progressive frames (as is yadif's wont, sadly)... because it never actually gets called on them. Same for hard telecine stuff that's been successfully IVTC'd. But it will catch things IVTC misses, as well as interlaced parts of hybrid content. It seems to take up around 5% of HandBrake's work loop load when it's running on progressive content, compared to 10% for vfr and 10% for scaling.
So, with pullup running first, followed by this comb detection and then one of the deinterlacers, I think we're approaching a one-click "set it and forget it" way of handling any NTSC content quirks. Of course, we could just run yadif all the time without checking if it's needed, but that's a lot slower and will lose detail on progressive frames. Plus, it should even save time on fully interlaced material, by skipping the deinterlacer during static shots. If we want to go for something grander, it seems to me that we could use a delay queue (expand the one used by yadif) and compare lines across frames.
http://pastebin.ca/896753
Code: Select all
Index: libhb/deinterlace.c
===================================================================
--- libhb/deinterlace.c (revision 1252)
+++ libhb/deinterlace.c (working copy)
@@ -84,6 +84,53 @@
hb_deinterlace_close,
};
+int detect_comb( hb_buffer_t * buf, int width, int height);
+
+int detect_comb( hb_buffer_t * buf, int width, int height)
+{
+ /* See if frame shows interlacing */
+ int j, n, off, block, cc_1, cc_2, cc, flag, eq, diff, thres;
+ uint16_t s1, s2, s3, s4;
+ cc_1 = 0; cc_2 = 0;
+ eq = 10; diff = 30; thres = 0;
+
+ block = width;
+ flag = 0;
+
+ for(j=0; j<block; ++j)
+ {
+ off=0;
+
+ for(n=0; n<(height-4); n=n+2)
+ {
+ s1 = (buf->data[off+j ] & 0xff);
+ s2 = (buf->data[off+j+ block] & 0xff);
+ s3 = (buf->data[off+j+2*block] & 0xff);
+ s4 = (buf->data[off+j+3*block] & 0xff);
+
+ if((abs(s1 - s3) < eq) &&
+ (abs(s1 - s2) > diff)) ++cc_1;
+
+ if((abs(s2 - s4) < eq) &&
+ (abs(s2 - s3) > diff)) ++cc_2;
+
+ off +=2*block;
+ }
+ }
+
+ // compare results
+
+ cc = (int)((cc_1 + cc_2)*1000.0/(width*height));
+
+ flag = (cc > thres) ? 1:0;
+
+ if(cc)
+ hb_log("flag: %i | cc: %i | cc1: %i | cc2: %i", flag, cc, cc_1, cc_2);
+
+ return flag;
+}
+
+
static void yadif_store_ref( const uint8_t ** pic,
hb_filter_private_t * pv )
{
@@ -520,13 +567,25 @@
{
avpicture_fill( &pv->pic_out, pv->buf_out[0]->data,
pix_fmt, width, height );
-
- avpicture_deinterlace( &pv->pic_out, &pv->pic_in,
+
+ if( detect_comb(buf_in, width, height) )
+ {
+ hb_log("found combing!");
+
+ avpicture_deinterlace( &pv->pic_out, &pv->pic_in,
pix_fmt, width, height );
-
- hb_buffer_copy_settings( pv->buf_out[0], buf_in );
+
+ hb_buffer_copy_settings( pv->buf_out[0], buf_in );
+
+ *buf_out = pv->buf_out[0];
- *buf_out = pv->buf_out[0];
+ }
+ else
+ {
+ hb_buffer_copy_settings( pv->buf_out[0], buf_in );
+
+ *buf_out = buf_in;
+ }
return FILTER_OK;
}
@@ -551,50 +610,59 @@
yadif_store_ref( (const uint8_t**)pv->pic_in.data, pv );
hb_buffer_copy_settings( pv->buf_settings, buf_in );
-
- /* don't let 'work_loop' send a chapter mark upstream */
- buf_in->new_chap = 0;
+ /* don't let 'work_loop' send a chapter mark upstream */
+ buf_in->new_chap = 0;
+
pv->yadif_ready = 1;
return FILTER_DELAY;
}
- /* Perform yadif and mcdeint filtering */
- int frame;
- for( frame = 0; frame <= (pv->yadif_mode & 1); frame++ )
+ if( detect_comb(buf_in, width, height) )
{
- int parity = frame ^ tff ^ 1;
-
- avpicture_fill( &pv->pic_out, pv->buf_out[!(frame^1)]->data,
- pix_fmt, width, height );
-
- yadif_filter( pv->pic_out.data, parity, tff, pv );
+ /* Perform yadif and mcdeint filtering */
+ hb_log("combing detected, passing to yadif");
- if( pv->mcdeint_mode >= 0 )
+ int frame;
+ for( frame = 0; frame <= (pv->yadif_mode & 1); frame++ )
{
- avpicture_fill( &pv->pic_in, pv->buf_out[(frame^1)]->data,
+ int parity = frame ^ tff ^ 1;
+
+ avpicture_fill( &pv->pic_out, pv->buf_out[!(frame^1)]->data,
pix_fmt, width, height );
-
- mcdeint_filter( pv->pic_in.data, pv->pic_out.data, parity, pv );
-
- *buf_out = pv->buf_out[ (frame^1)];
+
+ yadif_filter( pv->pic_out.data, parity, tff, pv );
+
+ if( pv->mcdeint_mode >= 0 )
+ {
+ avpicture_fill( &pv->pic_in, pv->buf_out[(frame^1)]->data,
+ pix_fmt, width, height );
+
+ mcdeint_filter( pv->pic_in.data, pv->pic_out.data, parity, pv );
+
+ *buf_out = pv->buf_out[ (frame^1)];
+ }
+ else
+ {
+ *buf_out = pv->buf_out[!(frame^1)];
+ }
}
- else
- {
- *buf_out = pv->buf_out[!(frame^1)];
- }
}
-
+ else
+ {
+ *buf_out = buf_in;
+ }
+
/* Copy buffered settings to output buffer settings */
hb_buffer_copy_settings( *buf_out, pv->buf_settings );
/* Replace buffered settings with input buffer settings */
hb_buffer_copy_settings( pv->buf_settings, buf_in );
-
- /* don't let 'work_loop' send a chapter mark upstream */
- buf_in->new_chap = 0;
+ /* don't let 'work_loop' send a chapter mark upstream */
+ buf_in->new_chap = 0;
+
return FILTER_OK;
}