Edelwises makes a good point about build quality. I had a Pedersoli 58 cal DR that was fun to shoot once I bedded the rifle. It looks like you can switch barrels and get different results. I cannot imagine both barrels having exactly the same, precise dimensions so that they seat identically. I would think you need to select one set of barrels as your primarily barrel set and bed the stock to those barrels.

93x64mm makes a good point in that any shifting of the rifle hold will change how a double rifle recoils.

I have seen one person shoot a double rifle and get small group and the second get big groups. As the rifle and loads didn't change, I believe that the difference he=ad to do with how tightly/firmly the shooters held the rifle. I think you are seeing this issue.