PM Modi's remarks came after former union ministers Arun Shourie and Yashwant Sinha criticised the government's economic policies.
"These pessimists are going global. Why don't they just go away?" Rahul Gandhi tweeted, using the IMF and World Bank reports to buttress his argument.
Mr Sinha and Mr Shourie had said the sudden withdrawal of high-value notes in November 2016 and introduction of the Goods and Services Tax or GST -- two major reforms of the Modi government -- have been badly timed.
Gross Domestic Product or GDP growth rate fell to 5.7 per cent in the first quarter this fiscal, the lowest in three years.
The IMF also attributed the fall in India's growth projection to the two big reforms. "In India, growth momentum slowed, reflecting the lingering impact of the authorities' currency exchange initiative as well as uncertainty related to the midyear introduction of the country-wide Goods and Services Tax," the IMF said.
The World Bank shared a similar opinion in its latest report.
PM Modi has defended his government's policies. "There are some people who sleep well only after they spread a feeling of pessimism. We need to recognise such people," he had said while addressing company secretaries on the Institute of Company Secretaries of India's 50th anniversary.
GDP growth rate had fallen below 5.7 per cent at least eight times when the Congress-led United Progressive Alliance was in power, PM Modi had said, adding his government was "committed and capable" of reversing the setback.