"Are you in politics or in wrestling?" Mr Yadav asked in Patna today, referring to Mr Modi.
Mr Yadav's inquiry comes after Mr Modi declared at a rally in Uttar Pradesh that "a 56-inch chest" is a pre-requisite to convert India's most populous state into one as developed as Gujarat. (Read)
Mr Modi's comment was targeted at Mulayam Singh Yadav, the leader of the Samajwadi Party, which governs Uttar Pradesh.
Mr Yadav's intervention is not surprising. In June last year, his party ended a 17-year alliance with the BJP over the latter's decision that its national campaign would be helmed by the Gujarat Chief Minister.
Mr Yadav and Nitish Kumar, the two big leaders of the Janata Dal (United), have alleged that Mr Modi is a polarizing figure who does not enjoy the confidence of minorities after the communal riots of 2002 that took place during his first term as Gujarat's chief minister.
An investigation ordered by the Supreme Court ruled that there was no evidence that Mr Modi was complicit in the violence, a charge levelled by his critics. That investigation was upheld recently by a court in Gujarat.