Rice has drawn heavy fire from Republicans for remarks she made in the aftermath of a September 11 attack on the US mission in Benghazi, Libya, in which four Americans were killed, including Ambassador Chris Stevens.
"I am highly honoured to be considered by you for appointment as secretary of state," Rice said in a letter to Obama. "I am fully confident that I could serve our country ably and effectively in that role. However, if nominated, I am now convinced that the confirmation process would be lengthy, disruptive and costly."
Her decision increases the odds Obama will turn to Senator John Kerry, chairman of the Senate Foreign Relations Committee and the 2004 Democratic presidential nominee, to replace Secretary of State Hillary Clinton when she steps down early next year.
Rice's surprise withdrawal spares Obama a potentially bruising nomination fight in the Senate, even as he grapples with congressional Republicans over tax, debt and spending policy to avoid the "fiscal cliff."
An announcement of Obama's national security team could come as early as next week. Officials say Obama is giving serious consideration to nominating former Republican Senator Chuck Hagel as his defense secretary to replace the departing Leon Panetta.
Obama had harbored hopes of picking Rice, 48, as the nation's chief diplomat. She was an early foreign policy adviser to him when he ran for president in 2008 and became the first black woman to serve as U.S. ambassador to the United Nations. She was widely seen as a natural replacement for Clinton.
But Rice has faced relentless criticism from Republicans about comments she made days after the attack in Libya.
She went on five Sunday television shows on September 16 to say that preliminary information suggested the assault was the result of protests over an anti-Muslim video that was made in California rather than a premeditated strike.
The video, posted on the Internet under several titles including "Innocence of Muslims," mocked the Prophet Mohammad and portrayed him as a womanizer and a fool.
'Strength of character'
US intelligence officials have since said that militants with ties to Al Qaeda affiliates were involved in the attacks, and acknowledged there were no protests. Rice has said she relied on talking points from intelligence agencies when she spoke, and Obama has angrily denied suggestions the White House played down terrorist connections for political purposes.
The White House has said Rice's comments were based on the best information she had at the time.
The Libya controversy, combined with a combative style and an eagerness to engage in partisan politics, undermined Rice's case, and visits she made to Capitol Hill in late November seeking to bolster her potential nomination did not help.
Obama, in a letter to Rice, said he was grateful that she would continue in her job at the United Nations. An aide said Obama and Rice spoke by phone earlier in the day.
"While I deeply regret the unfair and misleading attacks on Susan Rice in recent weeks, her decision demonstrates the strength of her character, and an admirable commitment to rise above the politics of the moment to put our national interests first," Obama said.
Rice and Obama are due to meet at the White House on Friday.
Republican Senator John McCain, Obama's 2008 presidential opponent, has been a strong critic of Rice and vowed last month to block any future promotion for her.
"Senator McCain thanks Ambassador Rice for her service to the country and wishes her well," said McCain spokesman Brian Rogers. "He will continue to seek all the facts surrounding the attack on our consulate in Benghazi that killed four brave Americans."
It was unclear why Rice made the decision now to withdraw her name from contention.
In an interview with NBC News, which first reported her decision, Rice said she did not want her candidacy to detract from the early months of Obama's second-term agenda, and priorities like immigration reform and deficit reduction.
"To the extent that my nomination could have delayed or distracted or deflected, or maybe even (made) some of these priorities impossible to achieve, I didn't want that," she said.
"Susan made this decision herself in the interest of avoiding a protracted partisan brawl," said White House spokesman Tommy Vietor.
Russian UN ambassador Vitaly Churkin, in a nod to Rice's reputation as a tough negotiator, jokingly told the PBS Newshour: "If it means that Ambassador Rice is going to spend another four years at the United Nations, I'll have to ask for double pay."
One option is for Obama to make Rice his national security adviser, a position that does not require Senate confirmation. But there has been no indication the current national security adviser, Tom Donilon, is leaving any time soon.
Kerry has long been seen as a possible successor to Clinton. He has friends on both sides of the political aisle and if nominated, is expected to win bipartisan confirmation easily after the new Senate convenes in January.
The Massachusetts Democrat in a statement called Rice "an extraordinarily capable and dedicated public servant." He added that "I've felt for her throughout these last difficult weeks" of political attacks.
As he molds a second-term national security team, Obama also has to name a CIA director to fill the position left by retired General David Petraeus, who resigned as a result of a sex scandal. Acting Director Michael Morell is a candidate for the top CIA job, along with White House counterterrorism chief John Brennan.
Also in the mix for the Pentagon job are Michele Flournoy, a former undersecretary of defense for policy, and Ashton Carter, the current deputy defense secretary.
Officials have discussed announcing the new national security team in a package. In another major appointment, White House chief of staff Jack Lew is a leading contender to replace Timothy Geithner as treasury secretary when he leaves after negotiations with Congress over US fiscal policy.