These two words both come from the Latin word
vertere 'to turn', but
averse (= turning away) means 'opposed to' and is typically used in negative contexts of people, whereas
adverse (= turning towards, hostilely) is used of things and means 'opposing one's interests; unfavourable' (
adverse circumstances, adverse weather conditions) or even 'harmful' (
the adverse effects of drugs):
The creation of a large source of illegal income has serious adverse consequences—Mirror, 2007
This proud, but humiliated, most complicated of politicians was not averse to flattery—M. Almond, 1992.