The Congress vice president, who is touring Gujarat, was labouring a point on the lack of women in the RSS but his reference to "women in shorts" provoked condemnation from the BJP.
"They are not interested in women's rights. How many women are in the RSS? Have you ever seen women in shorts working with RSS? I have never seen. You see women in the Congress all the time. In the RSS you will never see women. God knows what mistakes women have committed that they cannot be a part of RSS," Mr Gandhi told a gathering of women.
"Rahul Gandhi has used derogatory comments about the women associated with RSS. If he believes that wearing shorts in India is a symbol of empowerment, then I would like to condemn his statements," Ms Irani told reporters.
Another woman leader of the BJP, Nupur Sharma, tweeted: "If your objective is to see women in shorts there's little hope you'll be able to do anything for women empowerment."
Khakhi shorts were a trademark RSS uniform for drills, worn with pride by many BJP icons including Atal Bihari Vajpayee, LK Advani and Prime Minister Narendra Modi, before the organisation embraced a new look last year. The shorts have been replaced by brown trousers.
The change was reportedly driven by the concern that many young people were reluctant to join the RSS because of the flared, knee length shorts.