Usually, mentally ill folks (which can include drug addicts). I've known plenty of "fully functional" drug addicts that make life much harder for everyone, than some schizoaffective dude on the sidewalk.

Before the 1980s, when they started shutting down the "warehouse" mental institutions, homelessness was a far less pervasive issue.