It also raised hope that some oil ports occupied for 10 months by rebels in Libya's east will reopen. In April, rebels signed an accord with the government of Maiteeq's predecessor to unblock the vital Mediterranean ports but its implementation stalled when they refused to deal with Maiteeq, a businessman.
Port rebel leader Ibrahim Jathran welcomed the Supreme Court ruling, according to a statement.
Ahmad Maiteeq said that he would accept the court decision, which reinstates Abdullah al-Thinni as caretaker premier, according to parliament's deputy speaker.
Libya has had two premiers - Thinni and Maiteeq - with two cabinets since the latter got elected in a chaotic vote by parliament a month ago, compounding a sense of anarchy and drift three years after the uprising that overthrew Muammar Gaddafi.
Gaddafi's one-man rule over 42 years left Libya without credible governing institutions and security services to impose state authority on ex-rebels and Islamist militants, who now use armed muscle to carve out fiefdoms and make demands on Tripoli.
Thinni had originally resigned in April after what he said was a shooting attack on his family home by militiamen, but then refused to hand over power to Maiteeq pending a court decision.
"The ruling stated... the appointment of Mr Ahmed Maiteeq as premier of the interim government was unconstitutional," state television quoted the court as saying, without citing the legal basis of its decision.
Parliament's second deputy speaker Salah Makhzoum told reporters that lawmakers would respect the ruling. "Abdullah Al-Thinni is the caretaker prime minister until congress (parliament) learns the court's reasons for deciding Maiteeq's election was unconstitutional." Parliament will discuss the matter further on Tuesday, he said.
MAITEEQ ACCEPTS RULING
The General National Congress (GNC) is at the heart of a growing confrontation among rival political parties and brigades of former rebels who refuse to disarm and have allied themselves loosely on competing sides of the polarised legislature.
Those rivalries approached open conflict last month after a renegade former general, Khalifa Haftar, began a self-declared campaign with irregular forces to purge Islamist militants he says the central government in Tripoli has failed to challenge.
Fighters allied to Haftar stormed parliament briefly last month, accusing lawmakers of having no legitimacy and serving the interests of radical Islamists.
Maiteeq comes from Misrata, a western coastal city where the Islamist Muslim Brotherhood is strong but faces strong opposition in the east and in the western mountains.
In a brief statement Maiteeq said: "I do respect the rule of the Supreme Court and I am the first one who complies with the rule. What happened is devoted to the peaceful transfer of power and the first winner is the Libyan people."
Libya badly needs a functioning government and the reactivation of oil exports - the only notable source of state revenue - to prevent a wholesale collapse of state authorities.
Tripoli has no budget because the protests at oil ports and fields by militias and tribesmen making political and financial demands have reduced crude output to less than 200,000 bpd from 1.4 million bpd in July before the strikes started.
Libya has lost $30 billion from the oil strikes, a central bank official said last week.