The hold-up, at the Carlton Hotel on the promenade in Cannes, famous for its annual film festival, would be the second largest ever in France if the value of the jewels is confirmed.
Authorities said the robbery took place around 11:30am local time (0930 GMT) and targeted an exhibition entitled "Extraordinary Diamonds" by the Leviev jewellery house.
However, prosecutors later said the figure of 40 million euros should be treated with caution.
"The estimate of 40 million is not confirmed. It was given out in the beginning but an inventory is in progress. It's a preliminary estimate. It is not reliable," prosecutors from the nearby town of Grasse told AFP.
According to prosecutors, the gunman was wearing a "bandana or scarf" and a cap and was brandishing an automatic weapon.
He escaped with several satchels of jewels as well as diamond-encrusted watches, the prosecutors' office added.
"Everything happened very quickly and without violence," the prosecutors said.
Police detectives from the nearby city of Nice were at the hotel and traffic had been stopped near the scene of the heist.
The management at the five-star Carlton Hotel declined to comment when contacted by AFP, saying they had "instructions not to say anything."
The Carlton was the location for Alfred Hitchcock's 1955 Oscar-winning thriller "To Catch A Thief", starring Cary Grant and Grace Kelly.
The diamond heist was one of the largest in recent years in France.
In the country's biggest ever robbery, which took place in 2008, three men stole almost every piece on display at a jewellery exhibition in Paris with an estimated value of 85 million euros.
Cannes has fallen prey to thieves several times recently, notably during this year's film festival, which attracts a glittering array of celebrities from the movie world.
In a pre-dawn heist at a hotel during the festival in May, thieves stole jewellery worth $1.4 million due to be loaned to movie stars.
That robbery took place in the hotel room of an American employee of Swiss jeweller Chopard while she was out for the evening, police said.
In a scene straight from a Hollywood film, a strongbox containing jewels was ripped out of the wardrobe and carried off, they said.
In a second theft during the festival, robbers made off with a diamond necklace with an estimated value of $1.9 million.
At least two apartments rented by film executives were also burgled during the 2013 festival, with thieves taking cash, jewellery and other personal items.
And in 2012, thieves made off with four watches worth 400,000 euros belonging to football stars Souleymane Diawara and Mamadou Niang, who were in Cannes for the festival.
The world's biggest diamond theft occurred in February 2003 in the northern Belgian city of Antwerp, when robbers got away with jewels worth an estimated 100 million euros.