The proper grammar would be: "I am mad at you for ditching your
brother and me." "Me and your brother" works just as well. When in
doubt, just take everything out except the personal reference. You
probably would not say "I am mad at you for ditching I." or even
worse: "Me am mad at you for ditching I." I is used as a subject,
me is used as an object. The usage "I am mad at you for ditching
your brother and I" is a very common error made by people who are
perhaps trying a little too hard to be formal and correct.