HandBrake

I've been experimenting a bit with encoding h.265 videos with Handbrake (latest version at the time I write this..) and later edit these together with "TMPG Smart Renderer 5". If you're unfamiliar with that tool, it's a simple videoediting tool that allows you to merge/edit several videos to one file with minimal encoding, it just copies unedited parts to save rendering time..).
Anyway. That tool doesn't work very well with Handbrake encoded h.265 videos. It keeps insisting on recoding the movies even if it shouldn't be required.
I contacted their support and let them examine a file I've encoded with Handbrake, and their response was:
"You file has a long GOP.
It has a structure that the whole GOP1 ~ GOP3 has been recognized as a
Closed GOP Sequence. That which is counted as a single Closed
GOP of 1381 frames.

Since when the case of a MP4 target is smart rendered in Closed GOP
priority, it can not be cut from the middle of the Closed
GOP of 1381 frames, then the overall results have to be re-encoded.

When the number of frames of the source is recognized as more as Open
GOP rather than the Closed GOP Sequence, it does not occur such
phenomenon.
"
Does the above answer mean anything to you guys? If so, is there a setting to adjust this behaviour in Handbrake?

I tried encoding movies with Cyberlink Powerdirector in h.265 mode and this problem didn't appear at all, files were "smartencoded" without issues, so Handbrake is obviously doing something that Powerdirector is not (however I'd prefer to use Handbrake if possible ).

Their answer is kind of hard to decode (to me) but my best guess is that they're saying that the file they got had 3 GOP but that their code incorrectly didn't pick up that x265/Handbrake is using "Open GOP" and as a result their code decides that it's really ONE large closed GOP, presumably it does the same with your larger files with many more GOPs. Alternatively they could be saying that Smart Rendering isn't capable of handling Open GOPs at all but their final paragraph seems to say it does handle them when it recognize them.

The basic idea behind these kind of products is that they'll pass through any untouched GOP directly and just re-encode any frames in GOPs that are involved in a cut, but since it thinks there's only one very large GOP, it will always have to re-encodes it completely instead...

Obviously the best would be if they fixed their bug so it recognizes x265's Open GOPs correctly but until then you could likely work around it by telling x265/Handbrake to use Closed GOPs instead.

Using Closed GOPs will make the video larger at the same quality but I expect the effect to fairly small for the fairly long GOPs that x265 uses and I expect it'll still beat PowerDirector easily. According to the documentation I found on the net, adding "no-open-gop" to the "Extra Options" field ought to do the trick but I've not tested that.

You could also try the latest nightly Handbrake snapshot and see if it behaves any differently, it may or may not have a newer version of x265 which could in theory change things (or not).

This doesn't even say whether the input to TMPG (i.e. HandBrake's H.265 output) is MKV or MP4.

x265 has open-GOP enabled by default, and indeed, it may be helpful to disable it in this case ("open-gop=0" in the additional options text box, without quotes).

It could be that we're not signalling "keyframes" correctly in the case of Open-GOP, but TBH it's really tricky to get right (even mkvmerge had a related bugfix as late as a week ago, if I am not mistaken).

The "open-gop=0" doesn't fix the problem. Still lots of recoding being done.. Their tool work fantastic with h.264 content (made with Handbrake), but perhaps things will improve in the future with h.265.