The mass of the powder also contributes to the recoil. Even though it is converted to a gas, the gas still has the same mass as the powder it started out as. And there can also be an extra "rocket effect" where the gas will accelerate past the project as it breaks out from the bore.

Muzzle breaks divert the gas "mass" to reduce the recoil (change the direction).

If you have two loads with different powders but that give the same velocity for a given projectile, the one with the heavier charge will give more recoil (unless there is a big difference between the loads, the difference in recoil will likely not be all that noticeable).