I've read a lot of debates back and forth on this. Some say the same as you've expressed here about rather being at ISO 500 than 400 or 320 while others argue that this is a misunderstanding of the dynamic at play here.
For my part I can say this: I have been following the practice of trying to keep the ISO at 500 if it would otherwise be 400 or 320, but recently I have started to aim for the lower ISO regardless and at least my initial impressions has been that this is yielding better results. Why would that be?
Well remember: ISO does not cause noise: light levels cause noise (well, at least if we are talking about shot noise). We associate high ISO with noise because generally when we are using high ISOs it is when the light levels are low. This means something important: it means that if I require ISO 320 to get the correct brightness in my image that there is almost twice as much light as there is if I require ISO 500, and that means that there is going to be a much higher signal to noise ratio and less noise visible in the image. Yes, the ISO 500 will give me less read noise than the ISO 400, but it would make sense if the ISO 400 has less overall noise than at ISO 500 if I got to ISO 500 by increasing shutter speed or closing down the aperture and so reducing the actual light levels on the sensor.