I once heard in a documentary about steamboats on the Mississippi river that while the wealthy could afford to travel on steamboats, poor people could not and used small rafts
… that were called "rift rafts". From that term came the term "riff raff" meaning poor people and later "people with whom you did not wish to be associated". Is this story true? I don't know, but the documentary was on cable on either the science channel or the discover channel or other reputable channel. Perhaps someone else can confirm this etymology of the term. (MORE) 
