What is it called when monks or nuns self torture for god?

Mortifacation
"Doing penance" or "practicing mortification".