Should Kahlua be kept in the fridge?

No. Kahlua does not need to be stored in the refrigerator.