The presences of light fog indicates there was still moisture in the air despite the very cold temperature and even the act of breathing warm moist air over your rifle when bring it to shoulder could lead to condensation freezing around the firing pin and spring enough to lock it or slow it down. Even if the firing pin is driven forward, it doesn't take very much in slowing it down (lightning the blow) to result in misfires.

Even if all oil is cleaned out of the bolt and off the spring and firing pin, that won't prevent condensation that forms inside the bolt from freezing. Maybe a spray of de-icing fluid before a 'cold' hunt would help. This type of fluid gets rid of ice and prevents it reforming for a while.

Just tell the deer to hold on while you get out your spray can and de-ice the bolt!!!!!!