Roland Letendre, the current owner of the Berkshire Hathaway mill in New Bedford, started demolishing the structure on Monday after efforts to sell the property failed.
The mill, built in 1927, had been on the market with an asking price of $500,000, but it needs about $1 million in repairs, Letendre said.
"The building just needed too much work," Letendre told The Standard-Times of New Bedford.
Although it has been home to some businesses in the years Letendre has owned it, the mill currently has no tenants.
Letendre doesn't have plans to redevelop the site himself, but will likely sell the land, he said.
Buffett, then an ambitious 34-year-old investor, acquired control of the textile manufacturer in 1965. Overwhelmed by foreign competition, Buffett closed the mill in 1985 and sold the complex in 2000 to Letendre for $215,000.
Even though he expressed regret at buying the company, Buffett's Omaha, Nebraska-based conglomerate is still called Berkshire Hathaway Inc. Buffett is the nation's second wealthiest person, worth an estimated $58.5 billion, according to the latest Forbes 400 list.
Buffett once said he should have recognized the impending demise of the U.S. textile industry.
"The dumbest thing I could have done was to pursue 'opportunities' to improve and expand the existing textile operation - so for years that's exactly what I did," Buffett said in 2011.
The mill is being demolished despite efforts by Mayor Jon Mitchell to find a buyer and save what he considers a historically significant structure.
Buffett, apparently not sentimental, turned down an opportunity to repurchase the property.
Letendre emailed Buffett in October to tell him the building was available, but Buffett wasn't interested, according to story last month in The Wall Street Journal.
"I don't know what you'd do with that place," Buffett said.
Buffett did not immediately respond Tuesday to a message from The Associated Press about the mill's demolition.