Genetics certainly play a role in the end, but sticking with the size of the animals, the environment plays at the beginning a major role. On small islands, animals tend to get smaller. There were small elephants on many islands in the Mediterranean Sea.