Sources say, the final decision on what to make of Mr Verma's apology has been left to Mr Yadav. SP leaders however expressed and resentment and criticized the fact that Mr Verma only expressed a verbal apology.
Mr Verma had over the weekend accused Mr Yadav of having "links with terrorists". On Wednesday he expressed remorse after a meeting with the Prime Minister. "I am sorry if my remarks have hurt anybody's sentiments," he said.
The SP has made its fury clear by disrupting Parliament repeatedly, demanding an apology from the minister and his removal from the Cabinet.
The party has 22 Lok Sabha MPs, and props up the UPA government, which is in a minority. With the DMK exiting the Prime Minister's fragile coalition two days ago, Mr Yadav is now indispensable for the Congress-led UPA.
Sources say that the Congress has factored in the possibility of Mr Yadav ending his relationship with the government over the next few months. He recently urged his party to be ready for early elections.
At a meeting with MPs from Uttar Pradesh on Wednesday, Congress vice-president Rahul Gandhi reportedly issued a similar alert.
But Foreign Minister Salman Khurshid said that Mr Gandhi was not suggesting that the government will not complete its term. He said Mr Gandhi was emphasising that there are just months to go for the general elections, scheduled for next year.