Is it illegal to drive without a drivers side mirror?

In general I would say no, but you should check the motor vehicle laws in your state. I would consider outside mirror good safety equipment, but not something you need to operate the vehicle safely.

You would not have the missing mirror as an excuse if you have an accident, since it was clear it's missing and would require you to move your head around to see what the mirror normally helps you see.

The driver side mirror is usually more important (from a legal stand point) than the passenger side mirror. It depends on where you live, this is a law from NJ:

Every passenger automobile manufactured after January 1, 1965 and registered in this State, shall be equipped with an interior mirror and an exterior mirror on the driver's side.