Residents told Reuters the heavy-calibre shells hit a neighbouring residential district that is home to members of Assad's Alawite sect. State-run media said at least three people were killed and seven wounded in what it described as a "terrorist attack".
Syria's war, which has already spilled into neighbouring Lebanon and threatened to spread instability in the Middle East, poses one of the greatest foreign policy challenges for U.S. President Barack Obama as he starts his second term.
Fighters in the mainly Sunni Muslim opposition have stepped up attacks in the capital this week, setting off at least two bombs in areas populated by Alawites and assassinating two figures seen as close to the Assad administration.
The violence highlighted the sectarian dimension of a civil war that is deepening the rift between Sunni and Shi'ite Muslims in the region - Assad's Alawite sect is an offshoot of Shi'ite Islam.
An Islamist rebel unit said it targeted but missed the palace, a compound which sits on a hill overlooking the city that is mainly used for official functions. It was impossible to verify whether Assad was staying there at the time. He has several residences across the city.
"This operation came in response to the massacres committed in our beloved city," the Lions of Islam rebel group said in a statement. They said they also attacked a military airport and an intelligence facility in the capital, but there was no independent confirmation of those reports.
State television said a judge, Abad Nadwa, was killed on Wednesday by a bomb placed under his car.
Wednesday's attempted mortar strike on Assad's palace drew comparisons with a bomb attack in a highly guarded district of Damascus in July that killed four of the president's top lieutenants.
International and regional rivalries have complicated efforts to mediate any resolution to the conflict - Russia and China have vetoed three U.N. Security Council resolutions that would have put Assad under pressure.
The United States and other Western powers have also grown increasingly frustrated by divisions and in-fighting between Syrian opposition groups.
In a new push for unity, the main Syrian overseas opposition group the Syrian National Council (SNC) is meeting in Qatar on Wednesday to elect a new leader and executive committee.
A day later the SNC and other groups will meet to form a new 50-member civilian group that will later choose a temporary government for Syria and coordinate with the revolt's military wing.
U.S. Secretary of State Hillary Clinton said last week it was time to move beyond the SNC and bring in those "in the front lines fighting and dying" - a sign of U.S. impatience with a group that has long urged international intervention in Syria.
Smoke was seen rising from Mezze 86, the mainly Alawite area where the mortars fired at Assad's palace appeared to have landed.
"Ambulances are heading to the area and the shabbiha (pro-Assad militiamen) are firing automatic rifles madly in the air," said a housewife who asked not to be named.
A car bomb blast killed at least 11 people in the same district on Monday. Islamists, who are taking on an ever more prominent role in the war, also claimed responsibility for that attack.
State media said 10 more people were killed in a bombing on Tuesday in Hai al-Wuroud - another district that is home to many Alawites.
In what appeared to be a tit-for-tat response to those attacks, a car bomb went off on Tuesday in a working class Sunni area of the city that is a base for the rebel forces.
Air strikes and artillery barrages unleashed by the Syrian military in the last few weeks have wrecked whole districts of the capital, as well as parts of towns and cities elsewhere.
Yet, for all their firepower, Assad's forces seem no closer to crushing their lightly armed opponents, who in turn have so far proved unable to topple the Syrian leader.
The Syrian Network for Human Rights, an opposition monitoring group, said Assad's forces killed 154 people across Syria on Tuesday, mostly civilians in aerial and ground bombardment on Damascus and its suburbs, and the northern provinces of Aleppo and Idlib.
Lakhdar Brahimi, the joint U.N., Arab League envoy to Syria, said this week Syria could face the fate of Somalia, becoming a collapsed state where warlords and militias rule. The Syria war has killed about 32,000 people and left many parts of the country in ruins.
CAMERON WANTS ASSAD TO FACE JUSTICE
British Prime Minister David Cameron, who in the past suggested offering Assad immunity from prosecution as a way of persuading him to leave power, said on Wednesday the president would have to face justice.
"I would like to see President Assad face full international justice for the appalling crimes he has meted out on his people," Cameron said on a visit to Zaatari, a camp housing about 30,000 Syrian refugees in Jordan.
"I am standing with the Syrian border just behind me and every night 500 refugees are fleeing the most appalling persecution and bloodshed to come to safety and frankly what we have done so far is not working," he added.
The United Nations has put Syria's government on a "list of shame" of countries that abuse children, saying Assad loyalists have killed, maimed, tortured and detained children as young as nine.
Leila Zerrougui, special representative of the Secretary-General for Children and Armed Conflict, told Reuters on Tuesday the body was also investigating the opposition.