Thoughts on Automatic Cropping

[sterling_garnish]

I've recently had my share of overly aggressive autocrops, especially on very dark content like the Batman animated series, and on multi-aspect-ratio features, like The Grand Budapest Hotel. I know this behavior is the result of difficult tradeoffs, but I wonder if there might be a few tweaks that could improve the situation for most users without breaking anything.

Basics of the Issue

The current algorithm looks for the median size of black edges from the picture preview scans you see in the Summary tab, judging from previous forum discussions. The difficult tradeoffs here are real: a weak cropping algorithm would encode empty data and increase the risk of artifacts, while an aggressive cropping algorithm loses content. More sophisticated algorithms would be difficult to implement and test, and might hang.

While there's no perfect answer, judging from the frequency of over cropping to cropping artifact posts on the forums, taking the median value might not be the right compromise for most users. I personally worry more about losing content than introducing an artifact, because I can always shave off another column of pixels, but it's impossible to add back lost content unless I've also preserved the original source.

Possible Improvements

First, add a toggle in advanced preferences that could tweak the algorithm between two settings:

• Automatic (the balanced CURRENT BEHAVIOR)
• Conservative (looks for the MINIMUM safe value to crop across all frames, to tolerate a higher risk of artifacts but ensure no lost content)

I suspect a "minimum" setting would work well for many users, but in the interests of not breaking behavior, I think it would be better to start it as a non-default toggle that doesn't clutter the main UI. I considered a "MAXIMUM / AGGRESSIVE" option, that would really focus on reducing artifacts, but I just don't think that's actually a current need.

As a fallback, I would also increase the default sample frames slightly, even just to 15.

I'm less sure, but we might want to bring that option out from advanced settings and place it on the Summary tab next to the preview frames. People seem to not realize that option exists, but cluttering the main UI is risky. It might be easiest to educate users about the impact of that setting in the advanced preferences tab. In advanced preferences, directly beneath "Number of picture previews to scan:" we could add a comment:
(This setting will affect the accuracy of automatic cropping.)

Finally, I would also consider an offset for the very first preview sample frame by a few seconds. Often for films the Hollywood studio logo will appear and provide a good bright reference frame 3-5 seconds into the content. Right now the current preview sample frame is often completely black before the fade in, so it doesn't contribute useful information, visually to the user, or to the cropping algorithm. Not all content is a studio film, so this may be a hasty generalization, but it seems unlikely to worsen the first sample for most content.

Any thoughts on which option above would be the most effective? Any other ideas on even better ways to improve this balance?

Additional Reading

Back in Feb, rollin_eng recommended increasing picture preview scans to improve autocrop behavior:
viewtopic.php?f=7&t=38760&p=182822#p182823

In 2015, JohnAStebbins wrote:

HandBrake detects where these black bars are by sampleing several frames throughout the video and looking for the median values for the width of the bars. So if the width varies somewhat over the length of the movie, you will see some frames overcropped and some undercropped by a few pixels. There is no "best" fix for this scenario.

viewtopic.php?f=6&t=31519

[Deleted User 13735]

You see, the coders here volunteer their time. This isn't exactly their first rodeo (pun mostly unintended).
So it is unlikely the redesigned algorithms you "would" like to see "will" exist until "you" write them. Github is the best place for you to start.

[mduell]

Do you have a link to your proposed patch for review?

While there's no perfect answer, judging from the frequency of over cropping to cropping artifact posts on the forums, taking the median value might not be the right compromise for most users.

I think you comically underestimate the usage of HB if you think the scant forum posts about cropping are a big problem.

[sterling_garnish]

I've contributed to projects on GitHub before. In this case, I thought I might do some research on the forum, then ask people what they thought before diving in and offering a patch that breaks more than it fixes and is quickly rejected.

I was just trying to have a little humility and do some due diligence first.

Sorry if I'm wasting everyone's time and offending everyone.

[sterling_garnish]

It looks like they already implemented this, as "loose cropping." I'm not sure why it isn't the default, I can't find the rationale.

Code: Select all

        // don't try to crop unless we got at least 3 previews
        if ( crops->n > 2 )
        {
            sort_crops( crops );
            // The next line selects median cropping - at least
            // 50% of the frames will have their borders removed.
            // Other possible choices are loose cropping (i = 0) where
            // no non-black pixels will be cropped from any frame and a
            // tight cropping (i = crops->n - (crops->n >> 2)) where at
            // least 75% of the frames will have their borders removed.
            i = crops->n >> 1;
            title->crop[0] = EVEN( crops->t[i] );
            title->crop[1] = EVEN( crops->b[i] );
            title->crop[2] = EVEN( crops->l[i] );
            title->crop[3] = EVEN( crops->r[i] );
        }

[sterling_garnish]

More importantly, I can't figure out how to activate loose cropping in settings, it may be a dead option in the code. (This is not the same as the "loose" setting under Anamorphic that's nearby.)

[Deleted User 11865]

IIRC it was only ever exposed in the Linux GUI.