Maybe you are trying to compensate for the recoil too much and are pulling your shots. I try to just go with it and hold the rifle with a grip just strong enough to control the rifle.

I would do what JPK suggested and not do your shooting from a bench. Should help a lot.