It's normal that people don't care enough about you to figure you out before labelling you. It's more of an obesssive compulsive thing, that of HAVING to label and classify things and people. And the quick way is "if you're not with us then you're with them".
Not conforming, in school I was a communist, a fascist, a nihilist, basically I was "other".