So long as you don't use it as an excuse to become a recluse from or in society, you'll be fine. Believe it or not, none of us are all that alike- we share a lot in terms of interests, pursuits and hobbies, but we come from all walks of life and differing experiences. That is nothing to be ashamed of, in fact, you should revel in it.
When I was in school, all I ever wanted was to 'fit in'; being one of only a handful of Black kids in a predominantly white school district meant that I had not only racism but classism to deal with. What side of town you lived in or were born in made a big difference...
In any case, my desire to belong to something greater than myself arose from this basic need to fit in. I had to grow to be comfortable with myself, in my own skin before I could ever belong to something such as this- if I had spent my life up til now cataloging, recording and analyzing every slur or offhand negative statement about me or what I do, I'd be a really dark and disturbed individual.
Now, I have picked and chosen the people whose opinions matter; their opinions matter because I deem it so, because I care about them and know they care about me. All else though? Of no interest to me whatsoever.
I have found friends in some of the what I would've called the unlikeliest of places...