Typically, it finds its way home. The mechanisms it uses to do that depend on the exact species of ant.
Some rely on pheromone trails as their primary means of navigation; if these ants get lost, they’ll wander around, looking for the trail of other ants.
Some ants use visual landmarks when they’re far from their colonies. In one study, scientists led wood ants along a straight line next to a black wall to get a sugar reward. The wall would be on the left side of the ants as they walked toward the reward and on their right side when they walked back. The researchers then placed the ants midway along the wall; some of those ants had been fed, while others had not. The hungry ants kept the wall on their left, walking toward the sugar, while the ants that already ate walked back home.
Here’s the cool part: The researchers then placed the ants between two black walls. The hungry ants still walked closer to the right wall, and the well-fed ants walked closer to the left wall. That implies that the ants had formed memories, and that they were able to use those memories to navigate.
So, lost ants can use pheromones and visual cues. What about when they can’t those things?
Again, that depends on the species, but many will use a search pattern to try to find those all-helpful pheromones and landmarks, or other signals that could help them navigate home.
Like many other insects, ants are capable of tracking distance and direction traveled, and they can calculate a vector pointing back to their homes. When they lose this vector, they start traveling in small loops, gradually expanding the loops until they’re able to find helpful information.
We know that thanks to a 2017 study published in Scientific Reports, which compared the actions of experienced ants that had traveled a set route for several days with the actions of inexperienced ants (delightfully referred to as "naïve ants" in the research).
"At the unfamiliar test site, naïve ants ran off a longer portion of their vector from path integration than did experienced ants," the authors wrote. "Naïve ants also spread out in their systematic search slower than did experienced ants. We conclude that as ants learn the views encountered on their familiar route better, they identify more readily unfamiliar views."
In other words, most ants don’t rely on pheromone trails alone. They develop memories of their paths, then look for cues that will help them get home. Their search paths help them find those cues as quickly as possible.
All has failed. An ant can’t find its way back. What does it do?
Generally, they panic.
Again, the specific reaction depends on the species, but we do know that isolation is bad news for all types of ants.
One study found that when some ant species were placed alone in containers with adequate food and water, they died after six and a half days. They became hyperactive, searching for scent trails and visual cues that might lead them to their colonies—even though they had access to food, they kept looking for ways to get home.
"I think it’s somehow expected," Akiko Koto, co-author of that study, told The New Yorker. "In a natural condition, in the park or in a forest, if the ants lose their colonies, they’ll try to find their mother colonies."
If ants can’t find a way home, they eventually die from exhaustion or freeze to death. Like other insects, ants are ectothermic, which means that they’re dependent on external sources of body heat. Their colony provides that heat during cold nights, but if they can’t make it back to their colonies, they’re usually done for.