Most of the workers, including three block unit heads, resigned from the primary membership of the party in Maoist-affected Imamganj assembly constituency in Gaya district, considered a stronghold of Dalit leader Mr Choudhary, said a leader close to him.
This comes a day after Mr Choudhary, former Speaker of Bihar assembly, openly expressed his support for Rashtriya Janata Dal chief Lalu Yadav, the main political rival of Nitish Kumar.
Lalu Yadav is in a jail in Jharkhand's Ranchi after being convicted in a fodder scam case last week.
In last three months, Mr Choudhary has repeatedly attacked his own party and the Bihar government and targeted Nitish Kumar, who is also the party president.
However, JD(U) senior leader RCP Singh said the party will not issue notice to Mr Choudhary and there is no decision to take action against him. "It will be a wastage of paper to issue notice to him," he said.
In the last assembly elections in 2015, Mr Choudhary was defeated by former Chief Minister Jitan Ram Manjhi in Imamganj assembly seat after which he was sidelined.
But after Nitish Kumar dumped the RJD and the Congress to join hands with the BJP and formed government in July, he publicly rebelled against the chief minister.