Asking someone for the time of day is one of the most basic reasons for talking to someone, or the time of day is one of the most basic things that can be asked of someone, so to say that someone wouldn't give you the time of day is saying that he/she wouldn't talk to you under any circumstances.