Sean- the .416 Remington, a Rem 700, that I shot, had more felt recoil than my .458 M70Winchester which was putting 400gr. out at 2,450fps. To obtain 2,400fps, the smaller bore, but larger case requires more powder. Same bullet weight + same velocity + smaller bore size + greater pressure = more recoil. Yes- it has better sectional density than .458, but still kicks more. It has a slightly 'sharper' recoil, which may have been partly stock shape & fit. You would find the .404 and .458 easier to load to lesser, yet still deliver DG performance.
: I'd still go for the .404 - and load for it, of course.