In a ministry statement, Varoufakis said he and his wife had been threatened by masked anarchists at the working-class Athens district of Exarchia yesterday evening.
The anarchists threw objects at the couple, missing them, and told the minister to leave the area.
The minister described the incident as an "attack" but then argued that the group were not actually trying to hurt them.
After leaving the restaurant, Varoufakis said he spoke to the anarchists until tempers calmed.
But Prime Minister Alexis Tsipras this week reshuffled his negotiating team after Varoufakis reportedly irritated his fellow European ministers in successive meetings.
European economic affairs chief Pierre Moscovici has described Varoufakis as "not always easy" to work with.
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.