When did the earth's atmosphere start to form?

Depends on what you want to define as "atmosphere".

The first signs of a type of atmosphere occurred approximately 10 million years when the Earth started to form a cloud of gaseous silica.

When the Earth was about 40% its present radius, gravitational attraction would have been sufficient to retain an atmosphere which would included water.