I have asked the question about annealing 'old' new brass and been told that new unused brass that has been stored for a longish period should be annealed before use. I have some brand new RWS 404 brass (supposedly one of the best brands)still in original packaging and looking as good as the day it was made about 30 years ago but I will definitely anneal this brass before use.
Some of my early Kynoch factory 404 rounds crack necks on firing but the brass that doesn't crack, once annealed, lasts a long time for reloading.

Annealing when done properly will do much more good than any harm and is a very worthwhile and easy practice to give long life to valuable cases.

I've never heard of primers causing brass failure other than early corrosive primers and possible an old wives tale hangover from those days when some of those corrosive primers were hotter for the big cartridges (Elmer Keith mentions this in an article).

I think your problem likely a slightly larger chamber or neck area of the chamber in your bottom barrel but more important is brass that needs annealing before use.