IMHO only a chamber cast will show the real chambering of the rifle! Regarding the age I suppose it is rather a .450-400 2 3/8" BPE aka 10.3 x60R than a .450-400 3 or 3 1/4", if it is a British chambering at all. As you mention a deep rim recess and as it is a continental European make, it may be chambered for one of the countless cartridges on the so-called Mauser MB base. Regarding the "Serbian connection" of the maker, the 10.15x63R M1878 Serbian Mauser cartridge may be a likely candidate.
Jean van Malle is not listed in my sources on Belgian gunmakers, but apparently he had patented (brevete) a feature of this gun, perhaps the rebounding lockwork without half cock? Most likely the rifle was made by the specialized Liege guntrade for van Malle. European guns are usually signed by the retailer, often not the real maker.