Nobody can advise you how to deal with an issue that could concern safety, one can only speak for ones self.

I personaly would do such a conversion and shoot it myself without reservation if:
The action on examination was sound and tight.

If I wanted to be especially sure, I would have reciever hardness checked and the action magnafluxed or x-rayed.

Many .404's were built on standard length actions.

If one considers the likley point of failure, then with a modern moly barrel it is most unlikly to fail due to hoop stress unless in an overload/obstruction scenario.
So we are concerned with bolt/action lug sheer. A .416 Taylor working on the figures I have of Max allowable pressure equates to 56692lbs lbs thrust. The .404 equates to 49044lbs.
One can conclude from this that in the vital area likley to fail a 416 Taylor which will fit the standard length imposes a stress of 15% higher than the .404. If we need to remove by my measurments .060" of material to accomadate the .404, that is less than 15% of the material in that area and furthermore due to the angle of the feed ramp it is in fact much less.

All of the above is however, at the end of the day, for your interest only and is not me attempting to convince you one way or another, just giving you my best read on the situation.

Regards