At least one person died and three others were missing, a fire department spokesman said. According to authorities, 400 people were registered as living in the building, which belonged to the government.
The cause of the fire was not known.
Live television images in the early morning hours showed a firefighter on an adjacent rooftop talking to a man clinging to a rescue rope and trying to escape from the upper part of the burning building.
Suddenly, the structure collapsed and the man disappeared into the rubble. Authorities said he likely died, but were searching for him. In the afternoon, firefighters continued searching for additional victims amid concrete chunks and twisted metal pipes.
The abandoned former office building had 22 stories, according to the fire department and not 26 stories as reported by some media. It once housed a branch of Brazil's federal police and had been occupied by squatters for the past seven years.
Sao Paulo state Governor Marcio Franca, at the scene, told the Folha de S.Paulo newspaper it was a "tragedy foretold."
Franca said about 150 buildings in the region were occupied by organised groups of squatters, who have pressured the government for years to provide housing for the city's homeless.
The governor said it was legally difficult to force people to evacuate the old and decaying buildings.
"There is not even a minimal condition for people to live in there," Franca said. "People live there in desperation. This was a tragedy foretold."
The mayor of Sao Paulo, Bruno Covas, told reporters at the scene that the state government had offered to provide housing for the displaced families, and Brazil's president, Michel Temer, also offered federal assistance during a visit to the site.
Temer, a deeply unpopular president due to various corruption scandals and attempts to reduce Brazil's relatively expansive welfare net, left the scene quickly amid jeers from some of the displaced residents.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.