Well, it's your money after all.
Generally speaking if you're only one size off it can be fixed by getting another stem - a shorter one in this case. But if you're more than that off the risk is that you'll never get entirely comfortable on the bike.
And even if you manage to sort the comfort out the bike may always feel clumsy(for a bigger bike) or unstable (for a smaller bike).
Chat with our AI personalities