The main causes of sagging skin on the face with age are decreased production of collagen and elastin, loss of fat and muscle tone, and the effects of gravity over time. Sun exposure, smoking, and genetics can also contribute to the breakdown of skin elasticity, leading to sagging.