"We are no longer going to the EU-Africa. We disagreed on the composition of our delegation," said a source at the ministry, who asked not to be named.
Harare had earlier on Friday urged the African Union to shun the summit for failing to invite all the Africa bloc's leaders and lift a ban on Zimbabwe's first lady.
But diplomats in Brussels were unfazed by the call for a boycott.
"We see no risk" of a boycott of the April 2-3 summit, which gathers 90 nations from both continents, including 65 heads of state and government, said a senior official speaking on condition of anonymity.
In Harare, foreign ministry spokesman Joey Bimha said the European Union had failed to invite Sudan and the Sahrawi Arab Democratic Republic, which does not have international recognition, while Egypt, which has been suspended from the AU, had been given the nod.
Another concern is "the issue of our first lady who was denied a visa," Bimha told AFP, referring to Mugabe's wife, Grace.
Mugabe and his wife remain targeted by an EU travel ban but the restriction can be suspended temporarily to allow the head of state to attend international forums.
"We have been discussing this for some time. We have reached agreement and Zimbabwe has been invited but no spouses have been invited," the EU official said.
The EU ambassador to Harare, Aldo Dell'Ariccia, said when Zimbabwe asked for a visa for Mugabe's wife "they were told she should apply through the normal channels".
"The EU is just following it's legal framework and there can't be any movement from that position," said Dell'Ariccia.
Mugabe's spokesman on Tuesday said the EU's decision was "very strange".
"What God has put together the EU is trying to separate," said George Charamba in the state controlled daily, The Herald. "Do they expect the President to respect the EU and disrespect his own marriage?"
The EU has also refused to invite Sudan, although the Brussels official said the African Union was free to invite its president, Omar al-Bashir.
Bashir is wanted by the ICC for war crimes, crimes against humanity and genocide related to the conflict in the Darfur region of Sudan.