"I think Baghdadi's alive... and I'll believe otherwise when we know we've killed him," Mattis told Pentagon reporters.
"We are going after him, but we assume he is alive."
There have been persistent rumors that Baghdadi has died in recent months.
The Britain-based Syrian Observatory for Human Rights, a longtime conflict monitor, last week said it had heard from senior ISIS leaders in Syria's Deir Ezzor province that Baghdadi was dead.
Russia's army said in mid-June that it was seeking to verify whether it had killed the ISIS chief in a May air strike in Syria.
With a $25 million US bounty on his head, Baghdadi has kept a low profile but was rumored to move regularly throughout ISIS-held territory in Iraq and Syria.
The Iraqi native has not been seen since making his only known public appearance as "caliph" in 2014 at the Grand Mosque of Al-Nuri in Mosul, which was destroyed in the battle for Iraq's second city.