Why doesn't the gravity of the sun and moon affect us?

It does. Earth is moved by the gravity of the sun and, to a lesser degree the moon. We are move by almost the same amount and so we move with Earth. Since Earth's surface is our point of reference and we are moving with it, we don't notice it. We observe the effects of the gravity of the sun and moon in the tides.