"Whether we have to pursue Pakistan or put pressure, we will not rest until we get Dawood Ibrahim back," Mr Singh said in the Lok Sabha.
Hours later, Pakistan envoy Abdul Basit countered, "He is not in Pakistan."
The ruling BJP had been pummeled by the opposition after junior home minister Haribhai Chaudhury said in Parliament last week that the government has no idea of where Dawood Ibrahim is.
"The subject has not been located so far. Extradition process with regard to Dawood Ibrahim would be initiated once the subject is located," Mr Chaudhury had said, contradicting what successive governments have said for years.
The home ministry investigated whether the reply was the result of "drafting error." The minister clarified that he meant that India knew Dawood is in Pakistan, just not his exact location, as he kept shifting about.
Rajnath Singh told the Lok Sabha that Pakistan had failed to take any action against Dawood or track him down even after "all documents and overwhelming proof of his presence" there had been given by India.
Dawood Ibrahim, 59, is at number 8 on a list of 50 most wanted terrorists that India handed over to Pakistan in 2011. India has submitted several dossiers that it says are proof that he lives on Clifton Road in Karachi.