Discomfort with some people patterns or styles

I hope this doesn't get misunderstood by many here, I had few bad experiences with (let's call them people who try to trick others on streets - whether appearing as low level or not) - this made me avoid places that is not well organised or inhabited by good level people. Then I read in spirituality that we should show love to all and accept all? But I found that pushing myself to do so has a negative effect and keep me stressed all times which led me to a decision of not trying to act accepting all kinds and styles of people (note: it's mainly about feeling secure among them not as a matter of not accepting them in essence)... does this make someone arrogant, or far from spiritual path and fulfilment?