The move comes at a time when the AAP government and bureaucrats are at loggerheads since the alleged attack on Chief Secretary Anshu Prakash by some ruling party MLAs at the chief minister's residence in February.
"Exemplary action will be taken against the guilty in each case of corruption or irregularity pointed by CAG. No one will be spared (sic)," Mr Kejriwal tweeted.
Sources said that at least 50 cases from the CAG report have been identified by the government so far and the report is being studied for more such cases to be sent for CBI probe.
"At least 50 cases from the CAG report have been identified to be sent to CBI for inquiry. The report is being studied and the number of cases to be sent to CBI may increase. The chief minister himself is looking into the report," they said.
Earlier in the day, Deputy Chief Minister Manish Sisodia had said that those responsible for the lapses, be it a minister or any officer, will be punished.
In his address to the House on the CAG report, Mr Sisodia cited several "shortcomings and lapses" including construction of roads and public distribution system, and hit out at Lieutenant Governor Anil Baijal and bureaucrats.
"The CAG report has helped us in identifying rigidities in the system. We will remove the shortcomings and take action against those responsible, be it ministers or officers," Mr Sisodia said.