Panetta made the comments as the United States weighs how large a military force to keep in Afghanistan when the NATO combat mission ends in 2014, ending a war that, at that point, will have stretched for over 13 years.
There are approximately 66,000 US troops in Afghanistan, but the residual force may number less than 10,000. President Barack Obama could decide in the coming weeks, although no deadline has been set.
Commenting on the scope of a post-2014 counter-terrorism mission, Panetta did not mention al Qaeda allies or Taliban militants battling U.S. forces. He said fighting the core al Qaeda group to prevent it from re-establishing a safe haven in Afghanistan was "going to be the fundamental thrust of the (counter-terrorism) effort."
A narrow focus could help limit the size of the mission.
"Although we clearly have had an impact on (Al Qaeda's) presence in Afghanistan, the fact is that they continue to show up," Panetta told reporters at the Pentagon.
"And intelligence continues to indicate that they are looking for some kind of capability to be able to go into Afghanistan as well."
A US defense official, speaking on condition of anonymity, estimated there were still only about 100 Al Qaeda militants in Afghanistan.
But Jeffrey Dressler, an Afghanistan expert at the Washington-based Institute for the Study of War, said looking only at Al Qaeda fighters - as opposed to those who ally with them - carried enormous risks.
"I think the mistake that we've made all along is too narrowly defining the threat," Dressler said.
Beyond counter-terrorism, Panetta said the post-2014 U.S. presence in Afghanistan would also need to have a "train-and-assist mission" to further develop the Afghan Army.
He also said the United States would need to provide "enablers" - specialists who perform tasks such as destroying landmines or treating the injured - to support US forces.
Panetta declined to offer any estimate for the size of the force, saying that is "exactly what's being discussed" now.