First things first - make a good standing rest to test your rifle. You are taking an unnecessary beating from a .500 and not getting reliable results without one.

Light will affect where the rifle shoots quite dramatically with open sights and aging eyes. If it's coming from the side, the impact will be away from the light source.

Use OE 1 1/2F powder, it is the closest thing we have to C&H No.6 Nothing else has enough oomph. The 3F OE will give you over pressure loads. I've had it pressure tested.

Use a 340 grain PP bullet .503 naked - that's from an Eley drawing.

The .500 Lancaster thread has the load details. It will get you in the ballpark, you can adjust powder slightly from there if you have to.

Do not use NFB loads, the risk of ringing the chamber is too great. I have seen way too many of them.

BTW Lovely rifle!