The factory was among hundreds, where workers stitch clothes for Western retailers, targeted by safety experts for inspection in the wake of Rana Plaza building collapse last year that killed 1,135 people.
The team of experts ordered the evacuation of six floors of two adjoining buildings housing clothing firms Softex and Fame Knitting after the inspection last week which discovered serious structural flaws.
Softex, a sweater maker and a supplier to French retailer Auchan, shut down immediately, laying off all 3,500 workers, saying "it was risky to continue operations".
"I know what has happened to the workers is unfair. But the inspectors told me that the building has serious structural problems and asked us to evacuate the floors," Softex chief executive officer Rezwan Seilm told AFP.
Seilm said he was "desperately trying" to pay three months worth of wages to the workers from the factory in Mirpur in the northern suburb of Dhaka.
Fame, employing about 2,500 workers, also closed for a few days last week after engineers involved in the inspection requested reduced weight on some of the floors, an official said.
Unions criticised Softex and Fame for failing to compensate the workers, but a trade group representing garment factories said retailers must shoulder some of the costs.
"We're closing down plants as part of their (retailers') suggestions. So they must pay the affected factories to compensate their workers," said Shahidullah Azim, vice-president of Bangladesh Garment Manufacturers and Exporters Association.
But the sector has a woeful safety track record, highlighted by the collapse of the Rana Plaza in April last year on the outskirts of the capital, which killed 1,135 people, Bangladesh's deadliest industrial disaster.
The factory was the first to close following the mass inspections organised by dozens of Western retailers such as H&M and Benetton who signed a new safety accord following the tragedy.
As part of the accord, the retailers agreed to bankroll the safety inspections and loan the money for upgrades.
The head of the inspection team, Rob Wayss, said the factory has not been ordered to close down, but it has been asked strengthen the building's structural columns.
Wayss, who has been appointed by the retailers and global unions to look after factory inspections, said factories located on other floors of the twin-building containing 15 floors in total can continue operations.
"We've told the factories that they can operate in the remaining floors," Wayss told AFP. "We've also asked them to strengthen columns of the buildings."
Follow NDTV for latest election news and live coverage of assembly elections 2019 in Maharashtra and Haryana.
Subscribe to our YouTube channel, like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.