That one's easy - the 577 will give more velocity for a lesser bullet diameter and have a better BC, hence much more penetration. The 12-bore would need MV's quite a bit higher than the 577 to get equal penetration, and bullets quite a bit heavier. The 577 will also be better at longer range because of its better BC. Come on guys, don't tell me you never got out of the 19th century - and even the 1st half of it!?
The bore rifles, wile very good at short range, were found wanting at the longer ranges - because they didn't have the required penetration. If the velocity just isn't there, that big frontal surface won't get the job done.