Mr Baig made the remark in his address to Congress workers of Pulikeshinagar assembly constituency, which has a large Tamil population, in the city three days ago.
It stirred a huge controversy as the video clipping of his speech, delivered in Tamil, was aired by TV channels here today.
"When Modi was voted to power as Prime Minister, his supporters said he is our son. But, now what has happened? He banned Rs 1000 note. Banned Rs 500 note. Now, these very same people are chiding him," he said and went on to use an expletive.
Continuing further, Mr Baig said, "...it is not the Congressmen, but Gujaratis and Marwaris, who had supported BJP, who are saying this."
The remarks of Mr Baig, who is the urban development minister, drew flak from BJP with the party Lok Sabha member Shobha Karandlaje, charging it was only a manifestation of the "Congress culture".
BJP spokesman Madhusudhan said Modi was the prime minister not for BJP but for the entire country.
"Baig should mind his language. Otherwise, we will be compelled to hit back in the language he understands," he said.
Former Prime Minister and JDS supremo HD Deve Gowda said he disapproved of the expletive used by Mr Baig.
"Whatever maybe one's emotions....one should not cross limits...one should speak about issues and mind the language used," he said.