What is the number symbol called?

Most people just say "pound," but you can also just say "hashtag" (thanks to Twitter,) but you can always just say number sign.