Chris,
Nice looking rifle, so more pictures please!

Being "not sure" of bullet and groove could be a possible root of your issue:
"don't have great confidence" .....bullets are .502-.503 while lands-grooves are "perhaps" .504-.508, and you do not mention bullet hardness. (Very shallow rifling?)
Typical wheel weights/Lyman #2 or commercial hard cast grease groove bullets better be at least groove diameter.
You can get good results with undersized bullets with black powder if you use soft bullets that get slugged up by the BP when fired.
Talking about bullets cast of 1-30 or 1-20 tin/lead.

Paper patch bullets and cast of soft mix are an alternative if you are already familiar with the quirks of making and using them.

You would be miles ahead to get some no kidding accurate measurements of bullets and bore/groove dimensions then go from there.
- Mike