As counting went late into the night, the fractious multiracial opposition proved unable to unseat one of the world's longest-serving governments and pull off what would have been the biggest election upset in Malaysia's history.
The ruling Barisan Nasional (BN), or National Front, passed 112 seats, giving it a simple majority in the 222-seat national parliament. The opposition alliance had 60 seats.
With more than two-thirds of seat results confirmed, it remained to be seen if Prime Minister Najib Razak would receive a strong enough mandate to continue gradual reforms aimed at boosting investment and easing authoritarian laws.
He is under pressure to improve on the coalition's worst-ever result, recorded at the last election in 2008 when it won 140 seats and lost its customary two-thirds parliamentary majority. Failure to do so could weaken his leadership, unnerving investors and raising uncertainty over policy in the multi-ethnic nation of 28 million people.
"I hope the opposition accepts the result with an open heart and will allow the democratic process to continue," Najib told a news conference.
The opposition retained economically important Penang state as its leader Anwar Ibrahim sought to build on stunning gains in 2008, when it took over four state governments and deprived the BN of its two-thirds parliamentary majority.
It also claimed to have retained the crucial industrial state of Selangor, which Najib had vowed to win back, although that could not be confirmed with election officials.
The ruling coalition had been expected to win, but opinion polls showed a tightening race with Najib struggling to translate strong economic growth and a deluge of social handouts into votes.
The possibility of a disputed result loomed large amid opposition claims of widespread election fraud. Before most votes were counted, Anwar declared victory in a surprise statement that appeared to be a tactic to whip up support.
"PR has won," Anwar wrote on his Twitter account, urging the ruling party and the country's Election Commission "not to attempt to hijack the results".
Election officials said voter turnout was about 80 percent, a record high.
The campaign heated up in recent days with Anwar accusing the coalition of flying up to 40,000 "dubious" voters, including foreigners, across the country to vote in close races. The government says it was merely helping voters get to home towns.
The 2008 result signalled a breakdown in traditional politics as minority ethnic Chinese and ethnic Indians, as well as many majority Malays, rejected the National Front's brand of race-based patronage that has ensured stability but led to corruption and widening inequality.
Partial results from Sunday's election suggested that the trend of ethnic Chiense deserting the BN had accelerated, with the opposition Chinese party chalking up significant seat gains in the BN stronghold of southern Johor state.