I believe this came from terms that were used many years ago. The French traitors used to have their doors painted Yellow so as to define them. The old Sailors name for half caste was yellow belly and the Arabs felt it a sign of weakness if anyone was sick and brought up yellow bile.
It means they are a coward.
A coward
The colour associated with cowardice is yellow.
yellow fellow
Slang for coward-used in Westerns.
To be Asian or to be a coward, slang from the old west.
namby-pamby, yellow fellow
Saying that someone is yellow (a coward).
Such a person is usually called a 'coward'.
courage courage Its a coward or if there being racist... Chinese
When someone calls you "yellow", they are saying that you are afraid, craven, cowardly, etc.
He turned yelow means he became a coward or afraid of something.