I see your point, sort-of.
I'm pretty sure, if loaded to the same breech pressure, the difference might be much less than you think. Too, if all were loaded with 225 or 250gr. bullets, the larger bores would be slightly faster, if all were loaded with the same weight bullets, which would compensate for the lower BC's over normal hunting distances.