Question: What's up with fragrance?

I know that fragrance is bad and something I should avoid on ingredient labels. But, is it bad just because it could be masking phthalates, or for other reasons as well? A couple of products I've recently purchased have "Fragrance (Parfum)" listed on the label. I bought them because the labels also specifically said that the products were free of phthalates. So, my question is, if a product does contain fragrance, but does not contain phthalates, is it safe?

Here's an example of a product that's raising some questions. My general plan is to use up all of my products that I currently have and then replace them with non-toxic products. However, I didn't feel very comfortable with shaving my legs with my normal shower gel. (I feel that shaving, which strips off a layer of skin, makes me more vulnerable to ingredients in my shaving aids.) I purchased Alba Botanica natural very emollient cream shave in coconut lime.

The back of the label says that this contains "no: animal testing, artificial colors, parabens, phthalates, sodium lauryl/laureth sufate or sodium methyl sulfate."

I'm pretty happy with this shave cream. It clings to my legs well - I just slather it on, then shave. Usually my shower gels slides off and I use more that what's really necessary. It works just as well as shower gel; my legs are super smooth when I'm done. I checked on Skin Deep and it has a rating of 4... because fragrance is listed in the ingredients.

So, should fragrance always be avoided, even in phthalate-free products?