This makes sense.Of course distance matters, the more air you're trying to look through the higher odds that it'll be worse due to normal impacts, even if there isn't huge temp differentials, etc.
The sharper the lens to start with, the less it's impacted by this (but all lenses are).
It's not that I am suddenly getting great keepers of very distant tiny subjects where the zooms weren't doing this for me, but at least when I use the 500 I am definitely getting a much more defined edge and I can at least see things like feather details that are supposed to be there even if it's too small to really call "sharp."