How can you make your butt smaller and what are the exercises and the machines at the gym that you should avoid so you don't make it even bigger?

Generally, the machines go for high weight. You don't want to do that. You should go for low resistance, high repetition. Unfortunately, if you already have muscular glutes, you'll need to just let them go down from lack of use.