I suspect the answer would depend on what you want to wear under and/or over the vest. If you want this vest on the outside of a jacket or other thicker layer, you may want some extra room. If you think that you will only be wearing the vest and a t-shirt, and you are comfortable with a tighter fit, then you may like the small. But before you do switch, I would load up the vest and see how it all carries and fits. You might find that some of the extra room get used up by the gear.
Good luck,
--Ken