The minister, who was roasted on and off social media for his crude language, said he was only repeating what he had heard businessmen say in Tamil, a language that he claimed, he wasn't too well-versed in.
"I have highest respect and regards towards my Prime Minister. He's not a BJP Prime Minister, he's the Prime Minister of nation," Mr Baig, 66, said after the Congress called his remarks "condemnable".
Mr Baig was addressing a gathering in Pulakeshinahar, a suburb of Bengaluru Cantonment that has a large population of people of Tamil Nadu origin, three days ago when he made the derogatory remark.
The thrust of his speech was to highlight how PM Modi's most ardent supporters in the city had turned against him after he demonetised Rs 1,000 and Rs 500 currency notes last year.
"Earlier they would say he is our PM Modi... now they only abusing him," the Minister for Haj and Urban Development said, repeating the crude term that he claimed was being used by the business community.
As soon as the video got out, the BJP launched a fierce attack on the minister and asked Chief Minister Siddaramaiah to sack Mr Baig. State Congress leaders initially tried to defend him, others tried to suggest that the video had been tampered with.
Karnataka BJP President B S Yeddyurappa described the remark as "derogatory by all standards".
"They don't know how to use good language," BJP national secretary in-charge of Karnataka Muralidhar Rao said. "From Rahul Gandhi to Roshan Baig, why have they taken time to disown (the remarks). They talk about parliamentary system, but use unparliamentary language," he told reporters.
Later, he also added in a tweet that he had been targeted by the BJP because it does not have any real issues to highlight. Karnataka goes to polls in less than a year.
The Karnataka Congress had by then, already distanced itself from the comment.
Dinesh Gundu Rao, the party's working president in the state, called it "completely unwarranted".
"Speaking about anybody in such language, and more so, the PM is condemnable," he tweeted, asking leaders in responsible position to be careful about their choice of words. "We shouldn't lower the quality of political discourse," he said.
This isn't the first time that Mr Baig has found himself in the middle of a storm.
He had to quit from then Chief Minister SM Krishna's Cabinet in 2004 after his brother was arrested in the Karim Telgi stamp paper scam and there were whispers about his involvement too. His brother faced the trial and was eventually acquitted. Later, Mr Baig also faced a land-grabbing case.