What is the word for 'hat-maker'?

That would be "milliner". That is a word I haven't thought about in many years!

To be more precise, a "milliner" is a maker of women's hats. There is another term for a make of men's hats, but I can't recall the term. Anyone?

Yes, the term for a maker of men's hats is simply enough: hatter .

From inside a man's hat: Woodrow & Sons Hatters in Dublin

The term you were looking for is a "Modiste". ;)