Is that a fitting for a bayonet?

Last ditch defence against a wild boar perhaps?

I have French 12-bore "Cape gun" style shotgun. One rifled barrel, one smooth bore.

Is the "Cape gun" style gun common in France? What primarily was it used for? what sort of game?

So your gun is approximately
14b .693 in (17.6 mm) &
17b .649 in (16.5 mm).

You love your muzzleloaders.