As you're finding out mate - it's all about timing!
The bigger projectiles will go thru your bore at a slower speed & hence different impact point.
If you can shoot off a rest at 100m do so with it being in your hand, that way it will naturally let the rifle recoil instead of off a sandbag rest.