I can't tell from the photo what kind of scope mount you're using, but my experience indicates that the more stability you can achieve, the better the results. The Brookfield mount I am using (no longer manufactured) uses an Allen screw in the built in fixture on the side of the receiver, plus the dovetail for the clip loading adapter and a set screw on the receiver ring to anchor it as firmly as possible.

My M1A features a double lugged receiver and a heavy weight free floating barrel. It is a tack driver all the way out to 600 yards. It particularly likes Sierra 175 grain HPBT MK bullets. I once used it to shoot a 198-12X to beat all comers at the State Championship, including one former national champion shooting a bolt gun.