I believe this came from terms that were used many years ago. The French traitors used to have their doors painted Yellow so as to define them. The old Sailors name for half caste was yellow belly and the Arabs felt it a sign of weakness if anyone was sick and brought up yellow bile.
It means they are a coward.
A coward
yellow fellow
The colour associated with cowardice is yellow.
Slang for coward-used in Westerns.
To be Asian or to be a coward, slang from the old west.
namby-pamby, yellow fellow
Saying that someone is yellow (a coward).
Such a person is usually called a 'coward'.
courage courage Its a coward or if there being racist... Chinese
He turned yelow means he became a coward or afraid of something.
When someone calls you "yellow", they are saying that you are afraid, craven, cowardly, etc.