1/2 a thou is nice on the cases. Sounds as if the alignment of the seating stem is out of whack. Way out of whack.

On the other hand, Walt Berger once won a road trip BR match with ammo that was out .003". It was his bore fouling ammo with poor alignment (for BR ammo) and he'd forgotten to load his 'good' ammo.

Depending on the distance the bullet has to jump, it may be just fine - or not.

The general feeling, is the longer the bullet has to jump, the larger the chance it has to align with the bore and grooves.

Your sizing is perfect - I'd see about getting a Hornady seating die for 6.5. They are generic and will give straight line seating for all 6.5's.

I would even consider getting a 6.5x55 set and simply use them for neck sizing the 54's, which I assume you are talking about?