What is the origin of the phrase not to frighten the horses?

At the time when Oscar Wilde was on trial for indecent acts, Mrs. Patrick Campbell (a noted actress) was asked by a friend what on earth Oscar and is men frieds got up to, an wasn't it disgusting? Mrs. Pat replied, 'I don't care what they do, as long as they don't do it in the street and frighten the horses'.