Yes
James Madison.
James Madison warned against the dangers of what he called factions they produced "instability, injustice, and confusion."
Special interest groups and factions means the same thing but James Madison preferred to call it factions.
James Madison was for Majority Rule in Federalist Number 10. He was against factions that could work against public interest.
Federalist Paper no. 10 was written by James Madison to get the Constitution ratified. It discussed how to control factions that harbored interests that were in contrast to the rights of others. Madison argued on behalf of a strong government that could guard against factions.
Federalist 10
Political parties
James Madison wrote about his concern regarding factions in his famous essay, "The Federalist No. 10." He believed that factions, or groups of individuals united by a common interest or opinion, were a natural and inevitable part of human nature. Madison argued that the danger of factions lied in their potential to oppress the rights of others or undermine the public good, but that a large and diverse republic with multiple factions could help control their effects through a system of checks and balances.
James Madison worried about the concentration of power in the federal government and the potential abuse of that power. He was concerned about the threat to individual liberties and the possibility of the government becoming tyrannical. Madison strongly believed in a system of checks and balances to prevent this from happening.
James Madison believed that a democratic form of government, would tame the factions and cause them to work together as much as possible. Madison, in the The Federalist, argued that the federal system helps prevent factions from gaining too much control and causing tyranny of the majority
property
Madison argued that in an extensive Republic interest groups (factions as he called them) will be so numerous and varied that no one faction or group of factions will be able to control government and impose their will on the minority.