The two men were sent back on Friday over the killing of two Indian fisherman in February last year in a case that has strained ties between New Delhi and Rome.
"I resign in contention with the decision to send the marines back to India. The misgivings I expressed had no effect on the decision taken," Terzi told parliament during a special session dedicated to the contentious issue.
"For 40 years I have maintained, and still maintain, that the reputation of the country, the armed forces and Italian diplomacy, should be safeguarded," he said.
"I am also standing down in solidarity with our two marines and their families," Terzi told relatives of the pair who had gathered in parliament to hear the government report.
Massimiliano Latorre and Salvatore Girone, who are accused of shooting dead two fishermen off the coast of the Indian state of Kerala in February last year, were sent back to New Delhi on Friday after the government did an about-turn on whether they could stay in Italy.
The two, who were serving as security guards on an Italian oil tanker at the time of the shootings, claim they mistook the fishermen for pirates.
"It is not necessary now to go into whose fault it is... or bounce accusations of responsibility back and forth," the marines said about the furious debate within the Italian government over the decision to return the pair and their possible fate.
"What we ask from you now is not division but... to link arms, unite forces and resolve this tragedy," the men said in an email published by the country's main newspapers on Tuesday.
Italy has insisted the pair should be prosecuted in their home country because the shootings involved an Italian-flagged vessel in international waters, but India says the killings took place in waters under its jurisdiction.
Following a lengthy dispute, India's Supreme Court granted the two men bail to fly home to vote in Italy's general election last month -- on condition they were sent back to New Delhi within a month.
Italy agreed and its ambassador to India signed an affidavit taking personal responsibility to return them, which led to fury when Rome announced on March 11 that it was reneging on its commitment.
As the diplomatic crisis escalated, Indian authorities forbade the ambassador from leaving the country and airports were put on alert.
Italy said its change of heart came after it received written assurances that the marines would not face the death penalty.
The government said it had averted a diplomatic crisis, but the unexpected decision to return Latorre and Girone sparked anger in Italy, particularly as the men and their families had been told they could remain.
"I have always acted for the good of the marines and Italy. If I haven't managed that, I ask forgiveness from everyone, and first of all from both of them," Defence Minister Giampaolo Di Paola told parliament.
"It was me who told them about the decision to return them to India, I looked them in the eyes and told them," he said.
Di Paolo said he was not resigning because he had promised the marines he would not abandon them.
The two men had already been granted special permission once before to go home to Italy to celebrate Christmas with their families, before returning to India as planned.
On Friday, Italy's deputy foreign minister Staffan de Mistura urged India to move quickly to set up a special court to try the men, as proposed by the Supreme Court, and deliver timely justice.
Under a prisoner transfer agreement signed between the two countries, the marines could serve their sentences in Italy if convicted.