Citing lack of evidence, the Patna High Court last Friday acquitted Mr Yadav in the case related to Mr Sarkar's murder.
Left Front parties in Bihar last Saturday had urged the CBI to appeal in the Supreme Court against the acquittal.
"Yadav was released from the high security Beur Jail here," a police official said.
Hundreds of Mr Yadav's supporters had gathered outside the jail Tuesday morning. They shouted slogans in his favour and distributed sweets after he was released.
Mr Yadav's wife and former MP Ranjita Ranjan and several RJD and Congress leaders were present at the time his release.
Mr Sarkar was killed May 14, 1998, while returning from a meeting of party workers in Purnea.
A division bench of Patna High Court Justice VN Singh and Justice KK Lal set aside the February 2008 conviction of Yadav and co-accused Rajan Tiwari and Anil Kumar Yadav. The high court acquitted all three of all charges in the case.