Ben Carson, who is black, made the stunning remarks during an address to employees at the Department of Housing and Urban Development (HUD) in Washington.
"That's what America is about: a land of dreams and opportunity," said Carson, a retired neurosurgeon who grew up poor in a Detroit ghetto.
"There were other immigrants who came here in the bottom of slave ships, worked even longer, even harder for less," he said.
"But they too had a dream that one day their sons, daughters, grandsons, granddaughters, great grandsons, great granddaughters might pursue prosperity and happiness in this land."
The comments provoked an instantaneous backlash.
"Immigrants???" tweeted the NAACP, the nation's largest civil rights organization aimed at ending racial discrimination.
The remarks were condemned as "tragic, shocking and unacceptable" by the US office of the Anne Frank Center for Mutual Respect, a social justice group named after the Jewish girl whose diary, written before she was killed in the Holocaust, became a globally respected account of discrimination and hope.
"No, Secretary Carson. Slaves didn't immigrate to America," the group's executive director Steven Goldstein wrote.
"This is as offensive a remark as it gets."
The HUD department pushed back, saying on Twitter that the flurry of US media reports on Carson was "the most cynical interpretation" of his remarks.
"No one honestly believes he equates voluntary immigration with involuntary servitude," the department added.
It was not the first such controversy for Carson, a former Trump rival for the Republican presidential nomination in 2016 and someone who routinely blasts political correctness.
He once said Joseph, the Biblical figure, built Egypt's pyramids in order to store grain, and not as tombs for the pharaohs.
In 2013, he blasted the health care reforms of Trump's presidential predecessor Barack Obama as "the worst thing that has happened in this nation since slavery."
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)