In a gruesome report, they found that opposition groups in Syria's protracted civil war had also committed crimes of sexual violence and torture although these were "considerably less common".
The report also said ISIS and other armed extremist groups have executed women, men and children on charges of adultery, forced girls into marriage and persecuted homosexuals.
"It is utterly repugnant that brutal acts of sexual and gender-based violence continued to be perpetrated throughout seven years by most warring parties," Paulo Pinheiro, chairman of the U.N. Commission of Inquiry on Syria, told a panel event.
The 29-page report, issued as Syria enters its eighth year of war, is based on 454 interviews with survivors, relatives, eyewitnesses, defectors, lawyers and medical staff.
Karen AbuZayd, an American commissioner on the panel, said the documented cases represented the "tip of the iceberg".
Government forces raped civilians of both sexes during house searches and ground operations in the early stages of the conflict, and later at checkpoints and detention facilities, the report said. The youngest known victim was a nine-year-old girl.
"Rape of women and girls was documented in 20 government political and military intelligence branches, and rape of men and boys was documented in 15 branches," the report by the U.N. war crimes investigators said.
The independent experts - who have compiled confidential lists of suspects since 2011 - did not name individual perpetrators but said they had documented "numerous" cases of rapes by high-level officers.
Branches where rapes took place included Aleppo, Deraa, Homs, Hama and Damascus, as well as Sednaya military prison and the air force intelligence branch at Mezzeh military airport, both near the capital, the report said.
"Sexual violence against females and males is used to force confessions, to extract information, as punishment as well as to terrorise opposition communities," the report said.
Victims suffered shame, depression, incontinence, impotence and miscarriages, and rejection by their families, it said.
"Some interviewees have indicated it is worse for a girl to be raped than to be killed," AbuZayd said. "A number of women and girls sometimes commit suicide due to the verbal abuse they suffered in their homes or communities."
The investigators found "no evidence of a systematic practice" on the part of armed groups to use sexual and gender-based violence to instil fear, but said that incidents had occurred in the context of sectarianism or revenge attacks.
Throughout Syria's conflict, U.N. investigators had received allegations of extremist and terrorist groups imposing "mediaeval punishments on men accused of homosexuality".
"On ISIS, we recorded women being stoned for adultery, frequently for violations of the dress code, detention and beatings of women including pregnant women for minor violations, and forced physical examination to determine virginity," said Hanny Megally, a U.N. commissioner."We apologise for presenting such a starkly horrific report, but it important we get the information out," he said.