Curl is right - the 20 has fast twists meant for light weight .50 cal bullets in sabots - no comparrison.

Now, for pure smash, it might be a tossup between the .577BPE and a .725" RB at 10 to 20 yards, providing the RB had a descent load behind it, like 7 drams and up. In this case, the 12 bore RB's size would make up for slightly lighter weight and 100fps lower velocity, which is nothing. The .577, if hardened, might outepenetrate the hardened ball, but through and through is about all that's needed- generally.