It's time to name names here.

If the guy is asking for the $100 to try to cover his time up to now it's likely fair for you to pay him. Sure you should have arrived at an estimate or schedule of costs before engaging in a deal, and you own at least a portion of that responsibility.

Get square with him - and if you need mediation in order to simply make contact, stating his name here and now might lubricate that process. He deserves a voice in the conversation too anyway.

There's no doubt that the client/vendor relationship has soured. Learn from this experience and go forward, either to another builder or on a search for a complete rifle -- but do reflect and learn from this deal.

Cheers
Tinker