Gujarat Chief Minister Vijay Rupani had asked for Mr Patel's resignation from Parliament over the issue.
Reacting to the charge, the political adviser to Congress president Sonia Gandhi said his election to the Rajya Sabha had become an "eyesore" for the BJP. "When they face difficulties, they talk of nationalism and communalism. Recently, the chief minister said something about me, he should be ashamed (of it)," Mr Patel said, addressing a public gathering in the presence of Congress vice-president Rahul Gandhi.
"The way they weaved the story it seems that they have the sole contract over nationalism. Only they are nationalist, nobody else is," he said.
He said the Congress and its leaders had made many sacrifices for the country. "It started with Mahatma Gandhiji, and after independence, Indiraji (Indira Gandhi), Rajivji (Rajiv Gandhi). Many party workers and leaders became targets of terrorism... You cannot teach us nationalism, patriotism," he said.
"You cannot fight terrorism. You went and handed over terrorists in Kandahar (Afghanistan). How are you going to fight terrorism?" Mr Patel said, referring to the then NDA government's handling of terrorists in exchange for passengers following the hijacking of Indian Airlines flight IC 814 in 1999.
Mr Patel said there is no question of his resignation. "They are asking for my resignation. My win in the Rajya Sabha polls has become an eyesore for the BJP. I will not resign on your demand. My 44 MLAs have elected me," Mr Patel said, referring to the poll to the Upper House in which he had defeated his BJP rival.
The BJP recently accused Mr Patel of having "deep relations" with the hospital in Bharuch district.
"I had given (my) resignation from the hospital way back in 2015. I had joined it as the people of our area wanted us to build a good hospital. The management is different," said Mr Patel, who belongs to Bharuch. "Regarding the ISIS terrorist, I have done some research. He had earlier worked in two hospitals and one of them was inaugurated by BJP leaders. Why are there no allegations against them?" he said.