This is quite true in what you say, but you must remember that the punishment also applied to Cains seed. Cain was cursed with a dark skin as were all of his progenitors.
Since then Cains descendants have been a despised people who have been abused and segregated from all other races. Just as the Jews were and to some extent still are. Kamal Salibi (Secrets of the Bible People) says that the story of Cain is a conflation of at least four separate myths. That would explain the discrepancies. A slight glitch in the author's memory bank. It did not consider that there were only three of them (Adam, Eve, Cain) at that time.