At the velocities you are shooting the bullet spends some time in the bbl as it recoils and the muzzle moves up. Speed up the bullet and it will exit sooner while the muzzle is lower.

I dropped the velocity of my .458 WM to 1500 fps from 2000 fsp and it shot 15" high. I then reduced it to 1000 fps and it shot 3" high.

Weird, I know.