"L'ordre du jour" (in English "Agenda") had been among the favourites for the Goncourt prize -- the most prestigious in the French-speaking world.
Vuillard, 49, said he was taken aback on hearing he won for his elegant, 160-page book which charts how the financial support of German industrialists was crucial to Hitler's rise.
"One is always surprised, sometimes fatally," he told reporters at the Paris restaurant where the winner was announced.
Asked whether the book was a warning for our own populist times, the writer said he wanted to set out how the "elites and industrialists slid into a situation where they compromised themselves".
The Renaudot award, which is often seen as a consolation prize for those not shortlisted for the Goncourt, went to "The Disappearance of Josef Mengele" about the secret post-Holocaust life of the Nazi war criminal.
Journalist Olivier Guez tells how the SS doctor, notorious as the "Angel of Death" at the Auschwitz concentration camp for his often lethal experiments on prisoners, managed to evade arrest and even get a West German passport in his own name in the 1950s while living in Argentina.
Although his book was greatly admired, many critics had tipped Veronique Olmi to make history by becoming the second women writer in a row to win the century-old prize.
Her novel, "Bakhita", based on another real-life story of a Sudanese slave girl who became a Catholic saint, is already a bestseller. It has won the Fnac prize and is also in the running for the Femina prize, which will be decided on Wednesday.
"We did not choose the winner on the basis of the sex or origin of the writer, nor their publishers," said the doyen of French letters Bernard Pivot, who chairs the jury.
"The book is all that counts," he told AFP.