I think everyone has to make up their own mind, really. I don't enjoy listening to Michael Jackson's music because I associate it with accusations of pedophilia. Someone might not enjoy listening to Amanda Palmer because they feel her treatment of other musicians is exploitative, or that that her husband should be bankrolling her career as a musician. I don't know that Michael Jackson was a pedophile, and I don't know whether Amanda Palmer is exploitative of other musicians.
Sometimes a bad interaction can leave a bad taste in someone's mouth that spoils the art of a certain artist for them. I know a glassworker who thinks I am a royal-class asshole because on an anonymous forum (on which I was not anonymous, but he was) he private messaged me to tell me that people wouldn't be talking to him the way they were if they knew who he was. I asked him if he was seriously pulling a "do you know who I am?", said it was a dick move, and told him that I don't control anyone, and that he should expect people to respond to him based on what he says and does, not based on his status as an artist.
Dude hates me. And I no longer care for his art, which I used to admire. Art, including music, is something that evokes emotion, and if the emotion it evokes is unpleasant for any reason, including an irrational or unrelated association, people won't enjoy it.
Some people have opinions of Amanda Palmer that colors their enjoyment of her music. Others have completely different perceptions of her. LMNO can't stand her because of her fundraising, Waffles works with her and loves her. One isn't right and the other wrong; they simply have different experiences with her and interpretations of her actions.