M141-

I tend to keep my clothing light and tight, ammunition on the belt - even where it's cold but I don't hunt from a blind or a stand and am constantly on the move when in the hunt.
I have a couple of Filson wool coats that feature tubular pockets, covered by the pocket flaps on the chest.
I've seen guys use that kind of arrangement for spare cartridges at-the-ready.
Big pockets and pocket flaps on the upper chest of a coat or shirt can catch the butt of the rifle on it's way to your shoulder, cartridges there could make that worse if it affected you.
I've heard of guys taking small 2-cartridge belt ammo carriers and lashing them to their wrist with a velcro (or similar) watch band.
I've also seen some very slick and quick reloads from sling-mounted cartridge loops, the face of a cobra style sling can take two loops with room to spare (the sling for my .458 has a pouch that carries four)

Test-drive any setup you have time and time again every time you're at the range or in the field.
Do all your loading from the 'spare ammo' carrier when you're doing your target shooting or dry firing practice and get it down to second nature.

I've ditched a few shirts and light jackets that didn't end up working out for me due to issues with presenting the rifle or shotgun (pocket or flap catching on the buttplate or pad etc), done the same with ammo carriers and holsters that got in the way of getting the gun or ammo into action.

Cheers
Tinker