No, its not incredibly rough.
No you would not be unsafe there.
I think the stigma might be overrated.
But I am biased, I live nearby and buy quite frequently there for clients.
You will likely get all sorts of opinions from people. This has been discussed previously...