What is the area of science that study tiny particles like atoms?

It's kind of a broad thing. Individual atoms and molecules and their interactions with each other is the domain of chemistry. Interactions between subatomic particles, electrons, neutrons, protons, and even smaller with quarks and the like, falls under quantum mechanics (or quantum physics, as it is better known).