To bluff is to mislead by a display of confidence or claim of strength, and a bluff is an act of bluffing.
To call someone's bluff means that you think that person is not being honest, and you are challenging him to carry out a threat or to prove that his claim is correct.
Example sentence:
I don't think you have the nerve to do it, so I'm going to call your bluff.