In Virginia can you be charged with 'reckless driving and failure to yield' if all you did was fail to yield the right of way?

Yes. The basic definition of reckless driving is, "operating a vehicle in a dangerous manner". That being the case failure to yield to other vehicles could result in an accident and could be defined as reckless driving.