HandBrake

ok, I am starting with a raw mkv file that is 1920x800. For the simple reason of reducing file size, I like to reduce my resolution down to 720p HD. In my opinion, this is perfectly acceptable resolution and saves room I'd rather save for a better audio quality.

My question is, should I compress the video down to 1728x720 or 1280x533. I get that 1280x533 would be considered HD as the video would have black space above and below the video because its shot in a wider anamorphic view. But if I'm already starting with a higher resolution, which resolution would be preferable? Currently, I've been encoding down to x720 and letting the horizontal be dynamic to whatever the percentage says it should be.

Suggestion: Encode a movie each way, and play it on the device. Which looks better? Is the smaller one acceptable quality?

The more closely you match the device's native resolution, the better things will appear, GENERALLY. But that isn't guaranteed....

You're better off reducing the quality target and maintaining the full resolution.

If you reduce the resolution, I'd only do so to match your target playback environment's resolution, or if you're trying to get smaller than RF ~30.

Well, I put all my movies on a Plex server. I have many target devices from several tvs in the house, the tablets, to a phone, to friends who may stream from my Plex. I can't see just targeting a single device. I've adjusted my Handbrake settings several times over the years, mostly since the releases changes the settings around. I generally use RF20, reduce resolution to 720, and do 2 audio tracks, first AC3 6 channels 640 bit rate and a second AAC 2 channel to accommodate tablets and such.

I have never had any issues with quality since it seems like a good compromise between quality vs size (favoring quality), but just recently questioned how I was reducing the resolution. Any file size between 2.5gig - 4.5gig seems reasonable to me for a good quality movie.

You're better off staying at full res, and raising the RF to 24ish.

1280x533 is not HD unless it is encoded on a >=720 pixels high screen aspect. Player screen size doesn't mean jack.
Why are you needing smaller files? Drive space is running $20/TB.
Personally I would leave it alone at 1920x800 (2.4 aspect) and leave out the AC3 audio.

Gotta say, I am fdk-enabled, so take my last thought under advisement.