The move follows comments by Philippe Moreau Defarges about the outcome of the April 16 referendum on controversial constitutional changes that will tighten the president's grip on power.
Defarges, now a senior fellow at the French Institute of International Relations (IFRI), said all legal paths to challenge Erdogan had been shut off and that the only two options left were civil war or assassination.
The 'Yes' camp won the referendum with just over 51 percent, a narrower-than-expected victory, but Turkey's top election board last week rejected opposition calls to annul it after complaints of vote-rigging.
Defarges told French broadcaster BFM on Saturday that Erdogan's strengthened powers would lead "only to catastrophe".
"There will either be a civil war or another scenario his assassination," he said. He later apologised for the comments.
Huseyin Aydin, a lawyer representing Erdogan, said in a petition to an Ankara prosecutor that the comments were not a simple expression of opinion, but were "clearly instigating the crime in question", the state-run Anadolu news agency reported.
He said the comments showed how far the hostility against Erdogan had reached in the West, and suggested Defarges should undergo checks for his mental health if he ever came to Turkey.
If found to be in good mental health, his alleged links with the outlawed Kurdistan Workers' Party (PKK) or US-based Muslim cleric Fethullah Gulen -- blamed by Ankara for orchestrating the failed coup should be investigated, Aydin said.
Erdogan's spokesman Ibrahim Kalin confirmed the legal action, telling a press conference in Ankara: "We will do our best not to legitimise such fascist approaches."
'Test for Europe'
"Former French diplomat openly calls for assassination of President Erdogan. @IFRI_ should terminate his fellowship, apologise," Gulnur Aybet wrote on Twitter.
In a statement, the institute said Defarges spoke on a strictly personal basis, adding: "These comments do not represent IFRI in any way."
Defarges apologised Sunday, saying on Twitter: "Some of what I said was clumsy and might have been wrongly interpreted."
But Kalin said the apology was "not enough".
"This is not an issue that can be taken lightly," he said. "This is a test for Europe. Let's see how they will react."
Critics accuse Erdogan of becoming increasingly authoritarian after 14 years in power, and say the constitutional changes will lead to one-man rule in NATO member Turkey.
There have been numerous prosecutions for insulting Erdogan, with artists, journalists and schoolchildren all targeted.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)