Some say yes:
When you see an action actress that is muscular, you say to yourself that you would not want to get on her bad side. With today's action girls, you look at them and think sexy. This is the wrong image for an action actress.
Some say no:
I agree that they should look convincing, but that does not automatically equate tough or hard.
Matt Damon, a male action star, is very convincing in his parts. He is fit, but is not the old style classic cliche either. He is not built like Schwarzenegger or Van Damme, but in his parts, he can convincingly hold his own.
The same is true for female actresses. They do not have to be built like a bodybuilder in order to look convincing. Being beautiful or petite does not negate the fact that they are athletic or skilled.