Should you keep taking the pill if your doctor and obstetrician said not to worry about it?

It depends on what the doctors said not to worry about. If the pill is causing you side effects which you aren't happy with, then change your contraceptive pill. But if your not happy with taking a certain medication, always change if possible or speak it over with your doctor.