Bullet weight has the most influence on felt recoil. Two loads producing the same ME, but with light and heavy projectiles, will rock you way more with the heavy.

In the .458 WM for example. Both loads make 2700 ft/lb ME

1) 350 @ 1850 fps = 29 ft/lb recoil.
2) 550 @ 1500 fps = 40 ft/lb recoil.