Why did the prophet change his relation with the Jews?

Answer 1

It was because they did not believe he was a prophet.

Mohammed knew that the Jews believed in prophets, so he was counting on their support to set up the early Islamic community in Medina. If they accepted him as a prophet, the whole city would follow their example.

But the Jews decided almost instantly that Mohammed could not be a prophet. For the first year or so they maintained friendly relations, holding debates and challenging him to various readings of their scriptures. But when Mohammed asked them to make a definitive statement that they accepted him as a prophet, they replied that they did not believe in him.

Mohammed immediately changed the direction of prayer (qibla) from Jerusalem to Mecca and introduced new feasts, fasts and rituals deliberately designed to be different from Jewish ones. He stopped talking about Moses (assumed to be the ancestor of the Jews in Medina) and emphasised the importance of Abraham (assumed to be ancestor of the Arabs).

It was only a few weeks later that the Jews became very critical of Mohammed's policies. In particular, they perceived the Battle of Badr as a mass-murder and an inexcusable violation of human rights. Mohammed retaliated by attacking one Jewish tribe, the Qaynuqa, and driving them out of Medina. After this event, the Compact of Medina was definitively broken.
________________________________________________________
The answer above is falsifying facts. When prophet Muhammad migrated to Medina, he was fully supported by many and he was in no need to the support of non Muslims. However, he issued what is called Medina declaration that assured equal rights and obligations of all Medina citizens irrelevant to their faiths, colors, gender, or origins. However, Jews betrayed the prophet and Muslims and broke their promise to him to share with Medina citizens defending the city against any outside attack. One the city was attacked by the pagans (polytheists) they cooperated with them against Muslims. The Jews were sure that Muhammad is a real prophet and many Jews already converted to Islam before even prophet migration to Medina. They asked the prophet many questions that are known only to them and the prophet (by God support to him) replied them with the proper answers. So, they got confirmed that Muhammad is the true God prophet but they didn't like that the last prophet to be out of the Jewish community and to be from Arabs. See related link below.