Opponents fear the constitutional changes, which would grant Erdogan more power than any leader since modern Turkey's founder Mustafa Kemal Ataturk and his successor Ismet Inonu, would lead the country to one-man rule.
The 'Yes' campaign won 51.3 percent of the vote against 48.7 percent for 'No', the election commission said in figures quoted by state news agency Anadolu, in a count based on 99 percent of the ballot boxes.
In a nail-biting end to a frenetic campaign, the 'No' share of the vote climbed as more ballots were counted, after lagging well behind in the early count, but failed to overtake the 'Yes'.
"The presidential system, according to unofficial results, has been confirmed with a 'Yes' vote," Prime Minister Binali Yildirim told jubilant supporters from the balcony of the headquarters of the Justice and Development Party (AKP) in Ankara.
"This is a decision made by the people. In our democracy's history, a new page has opened."
But voting patterns showed Turkey deeply divided over the sweeping changes, with the 'No' vote victorious in the country's three biggest cities.
The 'Yes' vote held up strongly in Erdogan's Anatolian heartland but the Aegean and Mediterranean coastal regions and Kurdish-dominated southeast had backed the 'No' camp.
In a major disappointment for the president, the 'No' vote was just ahead in his hometown of Istanbul and in the capital Ankara and clearly ahead in the third city of Izmir.
In an interview with state TV late Friday, Erdogan had predicted a far clearer victory saying polls showed a 55-60 percent share of the vote.
'Vote for destiny'
But Turkey's two main opposition parties said they would challenge the results after alleged violations.
The pro-Kurdish Peoples' Democratic Party (HDP) said that whatever the final result, it would challenge two-thirds of the vote, saying: "There is an indication of a 3-4 percentage point manipulation of the vote."
The deputy head of the Republican People's Party (CHP), Erdal Aksunger, said it could appeal up to 60 percent of the vote.
"Believe me, this election is not over," he told CNN Turk, quoted by the Dogan news agency. "This is totally invalid. We are declaring this here."
The poll took place under a state of emergency that has seen 47,000 people arrested in an unprecedented crackdown after the failed putsch of July last year.
Closely watched on Monday will be the initial assessment of the international observer mission of the OSCE Office for Democratic Institutions and Human Rights (ODIHR) and the Parliamentary Assembly of the Council of Europe (PACE).
Beyond changing the government system, the vote could also have even wider implications for Turkey which joined NATO in 1952 and for the last half-century has set its sights on joining the European Union.
Erdogan has warned Brussels that in the event of a 'Yes' vote he would sign any bill agreed by parliament to reinstate capital punishment, a move that would automatically end its EU bid.
Western reactions to the referendum outcome will be crucial after Erdogan accused Turkey's allies of failing to show sufficient solidarity in the wake of the July 15 failed coup.
The new presidential system would dispense with the office of prime minister and centralise the entire executive bureaucracy under the president, giving Erdogan the direct power to appoint ministers.
The system would come into force after the elections in November 2019. Erdogan, who became president in 2014 after serving as premier from 2003, could then seek two more five-year terms.
Supporters see the new system as an essential modernisation step for Turkey that will remove the risk of the political chaos that blighted the 1990s and is blamed for the 2000-2001 financial crisis.
Opponents fear it risks granting Erdogan authoritarian powers and allow him to ride roughshod over key institutions like the judiciary and parliament.
Sinan Ekim and Kemal Kirisci of the Brookings Institution think-tank said in a report the changes if agreed "would set in motion the most drastic shake-up of the country's politics and system of governance in its 94-year-long history".
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.