The walkout by the far-right Lieberman raised the prospect that Netanyahu, whose conservative Likud party won the most votes in a March 17 election, will have to settle for a narrower alliance to secure a majority in the 120-seat parliament.
That could hobble the fourth-term premier, whose domestic policies are often resented at home while his championing of Israeli settlement in the occupied West Bank and East Jerusalem, lands the Palestinians seek for a state, is opposed abroad.
Netanyahu could ask the leading centre-left opposition party Zionist Union to join forces in a "national unity" government but both sides have so far played down any such possibility.
Briefing reporters, Lieberman said his party, which won just six seats in the election, had been offered two cabinet posts as part of the coalition talks but remained unsatisfied.
"This is certainly a coalition that, to my regret, does not reflect the positions of the nationalist camp and is not to our liking, to put it mildly," said Lieberman, adding that he was resigning as foreign minister.
Late on Monday, Netanyahu and the ultra-Orthodox Shas party signed a coalition agreement to raise to 53 the number of seats he controls. Last week the ultra-Orthodox United Torah Judaism (UTJ) party and centrist Kulanu signed on.
Likud is still negotiating with the far-right Jewish Home party which is fiercely opposed to the creation of an independent Palestinian state and wants to see the expansion of Jewish settlements on territory seized in the 1967 war.
Lieberman's Yisrael Beitenu party draws support from Israelis who immigrated from the former Soviet Union and has often come out against benefits for ultra-Orthodox constituents.
Lieberman complained that legislation anchoring in law Israel's identity as a Jewish state, which had been advanced by the outgoing government in the face of much controversy, was being played down in the current coalition talks.
Shelly Yachimovich, a senior Zionist Union lawmaker, did not rule out her party joining Netanyahu but said it appeared unlikely.
"I don't see an option like this," she told Israel's parliamentary television station. "It would be silly of me to consider something that does not exist."
Get Breaking news, live coverage, and Latest News from India and around the world on NDTV.com. Catch all the Live TV action on NDTV 24x7 and NDTV India. Like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.