If the EVF of the Z8 vs OVF of the D500 doesn’t bother you (I’ve found the Z9 EVF very DSLR-like with added benefits), I think you’ll easily adapt to the Z8. With the Z8 in DX mode, you’ll feel right at home, and you’ll have the added benefit of the full frame when you need it.
I typically shoot in FX and crop later. But if you’re used to DX, either shoot in DX on the Z8, get closer to your subjects or use longer focal length lenses. Note that in those instances where you can fill the frame in FX, you’ll have more pixels on your subject than can be done in DX, potentially giving you a higher quality image.
I wouldn’t say “should”. You now have a choice. If DX works best, leave it in DX. However, when the subject moves in too close, you can instantly “zoom out” FX by simply pressing a programmed button on the camera.
• How might the IQ from my 500mm PF lens be different with the Z8, if at all?
No different, and your AF with the 500 PF and a TC will be faster. Plus, AF is snappy in decent light even with 1.7x and 2x TCS.
They’ll be in the 50MB range in FX, but shooting in HE* RAW in FX, which is what I use, they’ll be the same as your D500. Also, shooting in DX mode in lossless RAW, they should be of similar size your D500.
I will warn you that the Z9/Z8 AF is very different than your D850. Some things are better and some are worse. But that’s a whole other subject.