Where does the term man of the cloth come from?

We start with the word "cloth," which may have been influenced by Clotho, one of the three fates in ancient Greek myths. Clotho was the one who spun the thread of life, which was then measured by Lachesis and cut by Atropos. The English word "cloth" does not directly derive from Clotho, but rather from a Germanic root of uncertain origin. It was clath or clathe in Old English, and the change of the vowel may have been influenced by the name of the Fate.
The next major step in the history is back in the days when royalty and nobility provided livery, namely, clothes for a servant. (Aside: The term "livery" itself derives from "delivery"; the clothes were delivered to the servant by the master. Another aside: The competition was so intense amongst the nobility for having splendidly dressed servants that Henry VII prosecuted nobles who gave liveries.)
From the elegant costumes of servants, other liveries arose--"clothes of calling," uniforms that became associated with certain professions. The term "man of the cloth" was thus originally applied to mean any livery, any clothes that distinguished a profession.
By the late 17th Century, the meaning had been restricted to the clerical profession, a minister or clergyman who wore professional garb. Jonathan Swift uses it in this sense in 1701, and so it has come down to us. The term can be applied to a minister or clergyman of any Christian denomination.
The word "clergy", by the way, comes from the Latin clericus (a priest) and from the Greek klerikos, which ultimately derive from the Greek keros, a piece of wood used in drawing lots. The word clerk and clerical have the same origin.