Several Congress leaders today tried to defend Mr Gandhi saying he had met the Prime Minister before flying out of town. (India Votes 2014: Full coverage)
"Rahul Gandhi met the PM before going abroad, there is no question of an insult," said the party's Satyavrat Chaturvedi.
But the murmurs that began at the farewell dinner have turned into a torrent of condemnation.
"The whole country knows how the Congress has treated Manmohan Singh. A PM for 10 years, when relinquishing his office, deserves a befitting farewell. Rahul Gandhi should have been there," said the BJP's Captain Abhimanyu Singh. (Sonia Gandhi Asks Congress to Defend PM As BJP Makes 'Scapegoat' Jibe)
"For 10 years you ruled this country, your PM ruled this country, if you could not take out time for his farewell dinner, then how will you take out time for the nation?" questioned Shiv Sena spokesperson Sanjay Raut.
Samajwadi Party leader Naresh Agarwal alleged, "Rahul Gandhi has insulted the PM. He thought of himself, above the country." (Read: On Prime Minister Manmohan Singh's Last Day at Work, a Standing Ovation)
Mr Gandhi's office issued a statement: "Rahul Gandhi met the PM on Saturday to say that he won't be in town. He thanked him already."
The clarification did nothing to stem the snowballing controversy. Congress ally Omar Abdullah, the Jammu and Kashmir Chief Minister, tweeted, "If you knew he was not attending the dinner AND he'd explained his absence to the PM earlier put out the statement BEFORE the news breaks. (sic)"
With exit polls predicting an unprecedented rout for the Congress, Mr Gandhi's absence raised speculation that he was reluctant to face questions on the party's performance. On Tuesday, several Congress leaders insisted that Friday's results would not be a judgement on Rahul, as he was not a part of the government. (NDTV Exit Poll: 279 for BJP and Allies, Congress in Double Digits)
As the BJP accused the Congress of making the PM a scapegoat, Mrs Gandhi stepped in to stop senior leaders from blaming Dr Singh and had the party issue a formal statement thanking him.
Get Breaking news, live coverage, and Latest News from India and around the world on NDTV.com. Catch all the Live TV action on NDTV 24x7 and NDTV India. Like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.