Mr Tata, who had stepped down as chairman and was replaced by Mr Mistry in late 2012, will head the group for four months while the company seeks a replacement.
Tata Sons is a large shareholder in a string of listed Tata Group companies - a business empire ranging from Jaguar Land Rover cars and steel mills to aviation and salt pans.
In a statement, the board said it was decided "it may be appropriate to consider a change for the long-term interest of Tata Sons and Tata group".
While the board gave no detailed reason for the change, some media reports said there has been discontent with some of Mr Mistry's actions, including asset sales.
The 48-year-old has been trying to shake up the $100-billion company by changing the management structure to bring in new faces at senior levels.
He has also battled issues on a number of fronts in recent months, including a costly settlement with Japanese telecom operator NTT Docomo and the sale of Tata Steel's loss-making UK business.
Britain's referendum vote in June to exit the European Union was a big setback for the steel sale, which Tata has now put on hold.
The Brexit vote has also cast a shadow over Tata Motors' luxury car unit Jaguar Land Rover (JLR), which has a large UK manufacturing base. Tata Motors' quarterly profit halved as the pound slumped following the Brexit vote, and JLR's CEO warned that some customers in Europe, its biggest market, no longer wanted to buy British cars.
Mr Mistry's ouster still took many by surprise, though analysts and investors saw Mr Tata's appointment as interim chairman as a way to ease concerns. "The impact will be a little softer with Ratan Tata taking over," Gaurang Shah, analyst at Geojit BNP Paribas, told BTVi television.
The board made its decision at a lengthy meeting on Monday, with six of the nine board members backing Mr Mistry's ouster, said one person familiar with the discussions. The person was not authorized to publicly discuss the matter, and did not want to be identified. Two members abstained. Mr Mistry, who could not vote, remains a board director.
Since stepping down as chairman in 2012, Mr Tata has pursued venture capital investments in ride-hailing firm Ola, e-commerce firm Snapdeal and payment solution provider Paytm.
International revenues make up close to 70 per cent of Tata Group's total turnover, and the majority of its capital spending has been overseas in the last three years.
The group's cash cows are Tata Consultancy Services and Tata Motors, together generating half the total annual revenue of $103.5 billion.