The men, aged 22 to 32, were detained on suspicion of being involved in the commission, preparation and instigation of acts of terrorism and of being members of the National Action group.
The neo-Nazi organisation became the first far-right group to be outlawed in Britain last year after the murder of member of parliament Jo Cox, whose killing the group had praised.
The four arrests were made by counter-terrorism officers in the cities of Birmingham, Ipswich and Northampton and in Powys, Wales.
"The arrests were pre-planned and intelligence-led; there was no threat to the public's safety," West Midlands Police said.
The Ministry of Defence (MoD) said a number of serving members of the army had been arrested.
Britain is on its second-highest threat level, "severe", meaning an attack is highly likely. Suspected Islamists have killed 35 people this year in London and Manchester, and a man died in June after a van was driven into worshippers near a London mosque.
Last month, a senior police chief said the number of referrals to the authorities about suspected right-wing extremists had doubled since the murder of Cox, who was killed in June last year by a loner obsessed with Nazis and white supremacist ideology.
(Reporting by Michael Holden; Editing by Stephen Addison)
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)