The far majority of those with carry permits are responsible, law-abiding citizens who do not want to have to ever use their firearm.
If someone does that, they're likely criminals (or capable of being) and damaged people in other ways too.
You can't deny 99.9% of responsible people who follow the law their rights because of the slim chance you'll run into someone who doesn't respect the law or other humans.
I can tell you, most who are "packing" are moral, law-abiding and properly trained. The few loose cannons and idiots give the rest of us a bad name. Of course, you're in Baltimore, a gathering spot for those types so you are justified in being afraid of something happening.
That's why I speak to no one in that city unless I'm ordering food at the Harbor.