Discussions revolved around India-Pakistan ties, asserted two ex-diplomats who attended the dinner.
Former Prime Minister Manmohan Singh and former Vice President Hamid Ansari were present at the dinner held last Wednesday at Mani Shankar Aiyar's home for former Pakistan Foreign Affairs Minister Khurshid Mahmud Kasuri's visit.
Manmohan Singh was present during the discussion but "did not say a thing", top sources have said.
The other guests included the Pakistan High Commissioner, former foreign minister K Natwar Singh, former Army Chief Deepak Kapoor, former diplomats Salman Haider, Satinder Lambah, TCA Raghavan, Sharat Sabharwal, Chinmaya Gharekhan and senior journalists including Prem Shankar Jha and Rahul Khushwant Singh.
"It was entirely devoted to talking about relations between India and Pakistan. Inevitably, cross-border terrorism came up and Kashmir came up but no one, I am 100 per cent sure, not a single person talked about either Gujarat elections or the domestic situation in India or for that matter, the domestic situation in Pakistan," Mr Gharekhan told NDTV.
Satinder Lambah, another invitee, said, "I don't comment on private dinners. There was just a general discussion on Indo-Pak relations."
PM Modi also said a former Pakistan Army officer has supported senior Congress leader Ahmed Patel becoming chief minister of Gujarat.
"(On one hand) Pakistan Army's former DG is interfering in Gujarat's election, on the other, Pakistani people are holding a meeting at Mani Shankar Aiyar's house," PM Modi said.
The Congress has demanded an apology for what it calls an outrageous charge.
"The Prime Minister must take back his words. He must respect his post by apologizing," said former union minister Anand Sharma.
"It is unfortunate what the prime minister has said about Manmohan Singh and Hamid Ansari. Former Prime Minister Manmohan Singh was respected the world over and his understanding of diplomacy is perhaps even better than that of PM Modi," Mr Sharma lashed out.