What is the key difference that causes the atoms of one element to differ from the atoms of all other elements?

The number of protons. This is given by the atomic number.