Mahathir's stunning defeat of the coalition that has ruled the Southeast Asian country since independence from Britain six decades ago means that, at the age of 92, he will become the oldest elected leader in the world.
Official results showed that Mahathir's Pakatan Harapan (Alliance of Hope) had won 113 of parliament's 222 seats, clinching the simple majority required to rule.
Najib's ruling coalition, Barisan Nasional (BN), had 79.
Mahathir told a news conference he expected to be sworn in as prime minister later on Thursday.
"The time for change has come, and I hope the people in power realise this," said Asifa Hanifah, a young woman who joined thousands of opposition supporters in central Kuala Lumpur who waved flags, cheered and honked car horns.
Few had expected Mahathir to prevail against a coalition that has long relied on the support of the country's ethnic-Malay majority.
However, he joined hands with his one-time protege, the jailed politician Anwar Ibrahim, and together their alliance exploited public disenchantment over the cost of living and a multi-billion-dollar scandal that has dogged Najib since 2015.
Mahathir has promised to seek a royal pardon for Anwar if they won the election and, once Anwar is free, to step aside and let him become prime minister.
Several key roads in the heart of the capital, where violence between races has played out in the past, were blocked off by police as evidence grew that Najib's coalition was on the back foot. The police appealed for calm and said in a statement that, for now, the situation was under control.
Najib's United Malays National Organisation (UMNO) party postponed an evening news conference and said Najib, who has ruled the country for nearly 10 years, would address the media at 9:45 a.m. (0145 GMT) on Thursday.
'HISTORY BEING MADE'
Malaysia's currency weakened in offshore trading on the election result, with the ringgit one-month non-deliverable forward falling 2.4 percent to 4.07 against the dollar.
The national stock exchange Bursa Malaysia said trading would be suspended on Thursday and Friday in line with public holidays declared by the government.
Stocks and the ringgit currency could take a hit on uncertainties over the newly elected alliance's fiscal and economic policies.
Ethnic-Malay Muslims have long tended to support BN for affirmative-action policies that give them government contracts, cheap housing and guaranteed university admissions.
Mahathir's alliance, which counts on urban votes and support from the minority ethnic Chinese and Indian communities, had hoped the veteran Malay leader would win over voters usually loyal to BN. That strategy appeared to have paid off.
"There has been a significant shift in the Malay vote," said Rashaad Ali, an analyst with the S. Rajaratnam School of International Studies in Singapore.
Mahathir's opposition wrested control of key states Johor and Kedah, and reduced BN's grip in strongholds like Sarawak.
Mahathir is a polarising figure and many voters are suspicious of him because of his iron-fist rule as prime minister from 1981 to 2003.
But 64-year-old Najib's popularity dropped sharply over the past three years, partly due to a scandal over 1Malaysia Development Berhad (1MDB), a state fund from which billions of dollars were allegedly siphoned off.
Mahathir was once Najib's mentor but he left UMNO over the 1MDB affair and joined the opposition. Najib, who was chairman of 1MDB's advisory board, has denied any wrongdoing and he has been cleared of any offence by Malaysia's attorney general.
In an even more unlikely change of heart, Mahathir buried a feud with Anwar, 70, last year and the two agreed to join forces to topple Najib.
Mahathir sacked Anwar as his deputy prime minister in 1998. Anwar then started a movement known as 'Reformasi' - reform - to end UMNO's race- and patronage-based governance. However, he was stopped in his tracks by charges of sodomy and graft, which he denied, but for which he was jailed.
Anwar was imprisoned again in 2015, when Najib was prime minister, after another sodomy charge, which he described as a politically motivated attempt to end his career.
(This story has not been edited by NDTV staff and is auto-generated from a syndicated feed.)
Get Breaking news, live coverage, and Latest News from India and around the world on NDTV.com. Catch all the Live TV action on NDTV 24x7 and NDTV India. Like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.