Based upon twist and weight I'd guess your Tolley is a conical bullet gun. Have you tried 750-800 grain conicals?

You say your round balls are gouping 2" apart at 50 yards. What happens at 100 yards? Assuming you have unaltered original sights, are the impacts high or low vs POA? Does relative accuracy of the barrels hold out? How far apart are the balls at that distance. I think you will find out what is really going on if you quickly move up to 100 yards for testing.

I'd go with Gatsby's suggestion for the 8 with wheel weight balls. You may find that undersized (.015 under) pure lead balls group better. I would not worry about aerodynamics of the round ball, we already know they stink.