How did Meccan rulers react to Muhammad's popularity?

Answer 1

Muhammad had only 150 followers in Mecca, so to say he was "popular" while he lived there would be incorrect. He was the leader of a small religious minority that was mostly ignored, save for its clear and unambiguous threat to destroy the religious property and paraphernalia of other religious groups. Meccan leaders properly took action to prevent that eventuality, but were overzealous in their attempts to assassinate Muhammad.

When Muhammad's religion became more popular in Medina, the Meccan leaders could not have cared less about Muhammad or his popularity. However, Muhammad began to attack Meccan trade caravans, pillage their contents, and murder their personnel. This wanton disregard for life angered the Meccan leadership which declared war on Medina when Muhammad refused to stop sanctioning this violence.

All in all, Meccans could not have cared less about how popular Islam was; they cared about how violent Muhammad was exhorting his followers to be. Thankfully, most of these exhortations are only preserved in the Hadiths or Sira, which allow for Muslims today to not follow them.

Answer 2

They tortured him and his followers to such an extent that they had to migrate from Mecca to Madina Munawara. Islam floured there and finally, just after 8 years, the Muslims conquered Mecca.