Is it a crime to open someone elses mail?

Usually. It's not a crime if they've given you permission to do so, and in some cases this permission is assumed: for example, in the absence of a clear intention to the contrary, spouses are usually assumed to have given each other permission to open each other's mail.