The number of people could range from three or four dozen to over three thousand. They included mostly unskilled workers, but also included others, depending on the type of castle. A stone castle would require a large number of masons, but regardless of type, woodcutters, smiths, and so on would be needed. The link below goes to a Wikipedia article, where you can find more detail.
It took about 456 people to build a castle
The number of people needed to build a castle can vary depending on the size and complexity of the castle. Historically, it could take hundreds or even thousands of laborers, craftsmen, and architects to build a castle over several years.
It took an estimated 5 years to build the Neuschwanstein Castle. Work started on the castle in 1868 and finished in 1872.
it took 20 years to build
Just as the case of many others, there is no record of the construction of Warkworth Castle. We believe it is dated from the middle of the 11th century, but do not have any idea how long it took to build.
It took 14 years to build Alcazar Castle.
it cost in pounds about 100 000 in them days that was more then hat people ever saw it took years to build 4 to be exact
Motte and Bailey castle took 10 years whereas when they upgraded it to stone, it took 40 or 50 years.
It took over 180 people
The Matsumoto Castle is the national treasure of Japan. The castles construction began in 1590 and ended in 1594. It took four years to build.
It took 600 people and 14 years to build the Brooklyn Bridge.
Chepstow Castle was built in fits and starts over the period of 1067 to 1300. I did not find any record that indicate how much of this time was actually spent on construction, but the impression I get is that it was not much more than 20 or 25 years.