The town contains many different kinds of people. There are the street urchins, young children who will mob an adventurer for money, and seem to come out of the woodwork when excited. Blubbering idiots are a constant annoyance, but not harmful. Public drunks wander about the town singing, and are of no threat to anyone. Sneaky rogues hang about watching for a likely victim to mug. And finally, no town would be complete without a swarm of half drunk warriors, who take offense or become annoyed just for the fun of it. (There are assumed to be other people in the town, but they are not represented on the screen as they do not interact with the player in any way.)
Most of the townspeople should be avoided by the largest possible distance when you wander from store to store. Fights will break out, though, so be prepared. Since your character grew up in this world of intrigue, no experience is awarded for killing the town inhabitants, though you may acquire treasure.