Have you tried 100 grains?
Have you tried a grease groove bullet?
Have you tried a bullet weighing 270 grains?
Have you tried a bullet weighing 365 grains?
Have you tried a greased felt wad between the over powder card and the bullet?
Are you using a chronograph?

While at the rest, are you holding the barrels and resting the hand on the rest, or are you resting the barrels on the rest?
How are you supporting the buttstock?

Is this Purdey rifle #9768?