I think rifle weight is as big a factor as stock design when you head north of the 450s.

We have a Sako BrownBear .500 Jeffery as our Big Game Rifle club-gun for Group 3 Nitro, and it is fine to shoot. Two other members have them also, and one very nice custom job on a CZ550.

Loads that make Group 3 are easy to shoot with these rifles, but add the extra 15 grains of powder to get them up to book velocity and you notice it.
Mind you, the .416 Rigby can sting a bit too with full loads, on account of the extra recoil velocity over a properly weighted .500.

Hope this helps: hard to put a subjective assessment like this into words.