HandBrake

When I output 4k content in h.264 from FCPX, the files look great on my media player (for exhibition). Due to the new players running h.265 I'm converting my files in handbrake, and have many settings choices to make. What settings (as many specifics as you might be troubled to list) would replicate what FCPX is default spitting out in the original h.264? I want to replicate the highest possible quality without making the files needlessly unwieldy for the Brightsign players.

Sorry for my ignorance, and thanks for any advice.

That's simply not possible with two vastly different encoders.
There is almost no correlation between settings and relative results.
If you're going to work in Handbrake, start with an appropriate preset, don't try to imitate FCPX h264, and feel free to share your workflow once your settled in with x265.

Forget about anything related to the settings for a different codec and encoder.

What are your playback devices capable of in terms of profile and level? Limit yourself to whatever the best is they support.

Beyond that, start with reasonable settings for speed/quality tradeoff (via the encoder preset system), tune appropriate for your content if any, and a quality target (RF). Look at the size of the output, the time the encode takes, and quality of the result and see where you need to go from there to suit your own personal preferences.

Thanks very much, musicvid and mduell, that shifts my understanding of the starting point.

From FCPX I export and save a 422 and an h.264.
To make my first 265's I converted 264's, mistakenly thinking they were similar.
Now I'm unsure - would it be better to start with the less compressed 422's?

From there, I'll make a small file and test some shifts from presets.

Yes, start the HB process from the highest quality master you have. ProRes 422 is a good choice, and supported.

The first step HB does is decoding the video to raw, so any prior encoding decision is irrelevant, just the quality of the decoded picture.

Makes sense. Thanks again for the help.