The BJP's Arun Jaitley called him a "delinquent officer" and accused him of showing blatant bias by refusing permission for a Modi rally at their preferred location in Varanasi. The party wants him removed as Returning Officer for Varanasi, from where Mr Modi is running for Parliament.
Arvind Kejriwal endorsed him today as a "good official."
Mr Yadav, 34, is the District Magistrate of Varanasi, which will vote on May 12, in the last round of the staggered nine-phase national election. He is an IIT graduate known to champion development.
Mr Yadav is said to be popular for building new roads and trying to remove illegal squatters in the temple town known to struggle with pollution and overcrowding.
He is believed to be working on a new master plan for Varanasi. During floods last year, Mr Yadav reportedly drew attention for working long hours.
After the controversy over the Narendra Modi events, however, the officer seems to have gained admirers and critics in equal measure, also on social media.
One section has described him as the "face of the fascist forces that are rattled by the Modi wave." Many in this group allege that he is close to the Samajwadi Party, which rules Uttar Pradesh.
Another set of tweets describe him as an "upright officer who worked tirelessly" during the floods in the city.