The announcement comes a month after the capture by Iraqi forces of second city Mosul further east in a major blow to ISIS.
In a televised speech, Abadi, dressed in military uniform and standing in front of an Iraqi flag and map of the country, announced "the start of an operation to free Tal Afar".
"I am saying to Daesh that there's no choice other than to leave or be killed," he said, using an alternative name for ISIS.
"We have won all our battles, and Daesh have always lost," he said, telling the country's troops that "the entire world is with you."
Tal Afar is located 70 kilometres (43 miles) west of Mosul, where US-backed government forces ended ISIS's rule in July after a months-long battle.
In June 2014, ISIS overran Tal Afar, a Shiite enclave in the predominantly Sunni province of Nineveh, on the road between Mosul and Syria.
At the time it had a population of around 200,000, but local officials said it was now impossible to know the exact number still living inside the city as most are cut off from the outside world.
However, authorities have accused the approximately 1,000 terrorists in the city of using civilians as human shields during Iraqi and coalition air strikes earlier this week in preparation for the ground assault.
Abadi said that Iraq's paramilitary Hashed al-Shaabi forces would help various army, police and counter-terrorism units in Tal Afar.
'Victory is near'
"In the early hours, the guns and flags turned towards their targets," said Ahmed al-Assadi, a spokesman for Hashad.
"Victory is near" in Tal Afar, an "Iraqi city taken hostage and humiliated for years by attacks from these barbarians," he said.
ISIS overran large areas north and west of Baghdad in 2014, but Iraqi forces have since regained much of the territory.
Once Tal Afar is retaken, Iraqi authorities intend to launch a fight to retake ISIS-held Hawija, in the province of Kirkuk, 300 kilometres north of Baghdad.
Terrorists still hold areas in Anbar, a western province that faces major security challenges.
ISIS, which declared a cross-border "caliphate" encompassing swathes of Iraq and Syria three years ago, has also suffered major setbacks in Syria, where around half of ISIS's de facto Syrian capital Raqa has been retaken by US-backed fighters.
But divisions across political, religious and ethnic lines will again rise to the surface in Iraq after the extremist group is driven out of its last bastions, experts have said.