I'd pay the $100 and be happy to get the action back and learn from the experience. It seems there were two different expectations here.

I am just about to pick up Custom Rifle #3 (from the same gunsmith) and if it shoots as well as the first two I will be a very happy man. I usually provide most parts and pay 40-50% of the final cost up front. Gets me higher up the queue too.

There is nothing like picking up the rifle when finally finished. It makes the costs, wait and anticipation well worth it.

You are getting your dream, not some manufacturers interpretation of what you need.