I think Weatherbys are just unusual rifles.
The cartridges are exactly what they were designed to be, Roy Weatherby was the "Viceroy of Velocity" after all, but the styling is what leaves so many people cold IMO.
I just never warmed up to the "California" styling - angled forend tips, flared grips, high cheekpieces, mother of plastic inlays and then the hard glossy finish. Goofy looking IMO.

I have also heared and read that the lock up of the little bolt lugs is simply not as positive as the Mauser system. Something to do with matching up all of the surfaces of the lugs evenly being very difficult resulting in few of them bearing the majority of the recoil. This may or may not be true and they may be designed differently now but I do remember looking at the uneven wear on the lugs of my father's 460 and wondering about it at the time. Maybe the more they are shot, the more even the mating becomes.