Three people were wounded in the attack, two of them critically, police said.
The incident was among the most serious in recent years in Jerusalem and was likely to heighten Israeli-Palestinian tensions.
No details were immediately available on the identity of the attackers.
The three were killed at the site known to Muslims as the Haram al-Sharif and to Jews as the Temple Mount, the location of regular clashes between Palestinians and Israeli police, but gunfire rarely occurs there.
The site includes the Al-Aqsa mosque and the Dome of the Rock.
A number of attacks have occurred around Jerusalem's Old City in recent months, but they have often involved knives.
Police locked down the area and the Al-Aqsa compound at the holy site was closed for Friday prayers. Gates leading to the site were sealed off.
Video being shared on social media appeared to show gunshots ringing out at the holy site.
"I was standing here and then I heard the shooting. I thought it was fireworks," Basem Badawi, a 60-year-old water seller in the Old City, told AFP.
"But then I saw the police coming from everywhere."
It is central to the Israeli-Palestinian conflict, with Palestinians fearing Israel may one day seek to assert further control over it.
It is located in east Jerusalem, occupied by Israel in the 1967 Six Day War and later annexed in a move never recognised by the international community.
Jews are allowed to visit, but not pray there to avoid provoking tensions. The site is administered by the Islamic Waqf organisation.
Waqf officials said its guards at the site had been detained by Israeli police following the attack.
A wave of unrest that broke out in October 2015 has claimed the lives of at least 277 Palestinians, 42 Israelis, two Americans, two Jordanians, an Eritrean, a Sudanese and a Briton, according to an AFP toll.
Israeli authorities say most of the Palestinians killed were carrying out knife, gun or car-ramming attacks.
Others were shot dead in protests and clashes, while some were killed in Israeli air strikes on the Gaza Strip.
The violence has greatly subsided in recent months.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.