So, I'm still not sure if fragrance is always a bad thing, but I will definitely avoid it from now on. My man and I each went to annual checkups this week. Both of our doctors advised us to stay away from soaps and other products with fragrance. In the words of my doctor: "People take showers thinking that they're getting clean, but frangrance stays behind in the skin and can cause irritation." Both doctors said that fragrance-free Dove is the best soap on the market. (This is a bit funny - my spray-on tanning salon specifically told me to stay away from all Dove products because they leave a film on the skin which messes up tans.)
From now on fragrance is something I'll be avoiding whether a product is phthalate-free or not.