HandBrake

So I've recently tried encoding my Bluray rip using the 10-bit h265 encoder. Question -- Why does this look superior to using an 8-bit encoder? I'm not seeing any color banding and the overall image looks "cleaner" with better color, but I always assumed that you should use an 8-bit encoder with 8-bit material. I may now want to re-do most of my library using the 10-bit encoder.

The reasons are complicated, but you are not imagining things. Banding is indeed reduced when using a 10 bit encoder with 8 bit sources.

Longer answer...

In encoded video, there are no "pixels". Instead there are frequency coefficients. Frequency coefficients are a representation of the image in the frequency domain, whereas pixels are a representation of the image in the spatial domain. When transcoding a previously encoded video, the video is decoded (transformed from frequency domain to spatial domain), then re-encoded (transformed from spatial domain to frequency domain).
When converting from spatial to frequency domain, the frequency coefficients are 32 bit values which maintains a lot of precision. But compression happens by throwing away some frequency coefficients (thus why it's called lossy encoding). Since some frequency coefficients were dropped, there will not be perfect reconstruction when converting from frequency domain back to the spatial domain (decoding). In addition, when you convert from frequency to spatial, there is a truncation to the pixel size (8 or 10 bit) that takes place. These losses are where the banding comes from.

Hmmm interesting. So is this doing a form of "color interpolation" and allowing smoother color gradients? It really does look so much better on my 85" Sony X900H in comparison, especially with shadow details. This is now inspiring me to redo many of my horror / sci-fi (which have a lot of dark scenes) and even my animated titles.