Which gas makes burning magnesium go out?

Nitrogen would extinguish the flame. Unlike most burning fuels, magnesium will continue to burn in carbon dioxide because its flame is hot enough to decompose carbon dioxide to carbon and oxygen.