In movies about the middle ages I've seen people sprinkle some kind of powder over their signature once they signed a document what was the stuff they used and why?

The ink they used in the pens, or rather the quills they wrote with back then was quite runny. Unless you used something to soak up any excess ink the letters would smudge and widen as the ink seeped further into the paper.

Penty of things could be used for that purpose, plotting paper, fine sand, etc.