Criticism from across the political spectrum was heaped upon still-unidentified city politicians from Prime Minister Shinzo Abe's Liberal Democratic Party, who unleashed the abuse during a city debate on child-rearing this week.
Chief Cabinet Secretary Yoshihide Suga, the government's top spokesman, cautioned that the incident in Tokyo's metropolitan assembly had nothing to do with national politics
But "if there were comments of a sexist nature, I would like the assembly to clean itself up," he told a regular press briefing in Tokyo.
Health minister Norihisa Tamura, whose portfolio includes the welfare of working women, said the abuse was "not only deeply disrespectful to women, it was a major human rights issue".
The comments came as colleagues of opposition member Ayaka Shiomura, 35, demanded the assembly identify and punish those who directed abuse at her.
Shiomura, from the centre-right Your Party, was questioning senior figures in the city administration on plans to help current and future mothers when she faced shouts of "Why don't you get married?" and "Are you not able to have a baby?"
She and her allies told local media that the widely-reported heckling came from an area occupied by local members of Abe's ruling LDP. None have been publicly identified so far.
Social media was also abuzz with criticism, while Tokyo's city office was inundated with telephone complaints.
"In order to show the public that these sexist comments cannot be tolerated, they should identify the people who made the remarks and let them take the heat," tweeted lawyer Ryo Sasaki, who added that the comments would mean a sacking in the private sector.
The Tokyo Metropolitan Government administers an area that is home to around 13.35 million people, about one-tenth of Japan's population, and has an annual budget of some 220 billion yen ($2.2 billion).
The city assembly has 127 members, of whom 25 are women. In national politics, women occupy just 78 of the 722 seats in the two legislative chambers of parliament.
The incident comes at a sensitive time for Japan's conservative premier, who has been pushing to boost the number of working women as part of his wider bid to kickstart growth in the world's number three economy.
Japan has one of the lowest rates of female workforce participation in the developed world and most economists agree it badly needs to boost the number of working women.
But a lack of childcare facilities, poor career support and deeply entrenched sexism are blamed for keeping women at home.