I am in kind of the same boat. Lots of rifles and have thought of narrowing things down.

Not sure why but I am an old school rimmed cartridge guy. As long as I could have a side by side 12 bore shotgun for birds and small game, I would take the following rifles for my needs:

Ballard 1885 High Wall in .300 H&H flanged
Searcy sidelock double in .470 NE

I never intend to hunt elephant, if I did I might opt for a .577 NE in the latter.

It is fun to think about, but having lots of guns in the safe is fun too....