Is it considered abuse if your brother and his wife keep telling you that you are 'too dumb to go to college'?
Not really, because I don't think they know how good you are doing in school. Have you ever tried saying how you feel about them telling that to you. Don't try to call them bad things or they will only be discouraging you more that you are not smart. Also try telling your parents about this and they might talk to your brother and his wife.
NO ONE has the right to tell you even in a nice way your chances of making it in college are slim. If you have the dream to go and work hard at it there is no reason you can't succeed as long as you have graduated. Some of the most famous people were told they couldn't do what they wanted to do and they succeeded. You don't need permission from your brother and his wife to go to college if you are not a minor. I was 55 years old when I entered into two courses in college and my friends thought I was crazy, but I went, got good grades, made good friends (much younger than I did) and I felt great and very proud of myself.
The Wright brothers were told they'd never get their plane off the ground. Amelia Earhart was scorned for being a woman pilot and she jumped the hurtles of negativity. Years ago when science was young many people scorned a scientist that they were "off the track" on whatever they were endeavoring to help mankind. Thank God these people didn't listen to the downers in this world.
It's much better to have tried than not tried at all. I have a dream of making my book into a script and my chances of having it accepted is low, but hey, I'm trying and one never knows. I've enjoyed the journey no matter what!
Obviously they have a blind spot regarding your ability to make things happen. Let the comments blow over without a reaction and don't keep them abreast of your school activities. As I used to tell my sons, the other people don't live your life--you do, so follow your heart. Go to college--you will gain confidence and learn things you never thought you would about yourself.