Ironoxide your rifle must be recoiling in a certain fashion to throw its pills either left or right, there may be a possibility that shooting your left & right barrels (individually at separate targets) you are moving the rifle on your shoulder (different angle on your shoulder )& hence doing so without realising.
Just a mad theory.
Consistency is the key as you are finding out, can you try left & right barrels at one target only; best to get a mate & call your shots as you go.
Keep trying mate, I think though you aren't too far off licking this problem!