The 12-storey hotel was evacuated and crowds quickly built up outside as around 120 firefighters and 20 fire engines tackled the blaze on the roof.
The five-star hotel completed what it called the "most extensive restoration in its 115-year history" only last week, after nearly two years of work.
With almost 200 rooms, the hotel is located near Harrods department store in Knightsbridge, an upmarket area of central London.
Firefighters were called to the blaze at 3:55pm (1455 GMT), and were expected to continue working into the night.
"The smoke from this large fire was very visible across central London," said fire brigade station manager Pam Oparaocha.
"Once the fire has been brought under control fire investigators will take over to establish how the blaze started."
A spokesman for the hotel said they would provide an update as soon as they could.
Roads were closed off near the hotel, an Edwardian-style building that opened its doors in 1902 and was bought by Mandarin Oriental in 1996.
Penelope Evans, 50, from Battersea in south London, was leaving Harrods when she noticed the blaze.
"It was like people were in shock, everyone immediately just started leaving the street and the area."
Firefighters could be seen crowded on a fire escape outside the building.
There appeared to be no visible damage to the outside of the hotel.
Transport for London, which manages the city's public transport system, said an entrance to Knightsbridge Underground station had been closed "at the request of the emergency services".
The London Ambulance Service said had sent incident response officers and a hazardous area response team to the scene.
"We have not treated any patients but we remain at the scene as a precaution," it said.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
Election Results for Lok Sabha Election 2019 will be out on May 23. Get the latest election news and live updates on ndtv.com/elections. Catch all the action on NDTV Live. Like us on Facebook or follow us on Twitter and Instagram for news updates from each of the 543 parliamentary seats for the election 2019