I had a similar thing happen with my .470. At first I thought the gun was shooting about 3 inches low and considered steps to change the front sight, just as you did. It was then that I realized that maybe I was aiming the gun with the front sight bead too low in the "V" of the rear sight. By adjusting the way I used my sights and now positioning the bead higher in the "V," the gun shoots dead on.
It took some adjusting on my part after a lifetime of aiming a rifle with iron sights a certain way, but after plenty of practice, I now shoot my .470 just fine. It's simply possible that the maker of your gun positioned his bead differently than you do. In your case, maybe you aim with the bead high in the “V”. If you try aiming with the bead at the bottom of the “V”, it might be all you need to do to lower your POA and get the accuracy you want.