Pyongyang in recent months has sparked global alarm by conducting a sixth nuclear test and test-launching missiles capable of reaching the US mainland, while Trump and the North's young ruler Kim Jong-Un have traded threats of war and personal insults.
Mattis, on a trip to Seoul for annual defence talks, maintained that diplomacy remained a "preferred course of action" but stressed, "our diplomats are most effective when backed by credible military force".
"Make no mistake -- any attack on the United States or our allies will be defeated," Mattis said at a joint press conference with his South Korean counterpart Song Young-Moo.
"Any use of nuclear weapons by the North will be met with a massive military response, effective and overwhelming," Mattis said, adding Washington "does not accept a nuclear North Korea."
"I cannot imagine a condition under which the United States will accept North Korea as a nuclear power," he said.
Mattis did not specify the threshold of nuclear weapon activity that would trigger a military response. Pyongyang's Foreign Minister Ri Yong-Ho said on the sidelines of the UN General Assembly last month that his country could test a nuclear bomb over the Pacific.
But Mattis said Pyongyang should "harbour no illusion", saying the isolated state is militarily "overmatched" by the US and South Korea -- a key ally of Washington that hosts 28,500 US troops.
'Not rushing to war'
Mattis' trip comes ahead of Trump's first presidential visit to South Korea next month as part of his Asia tour which also includes Japan, China, Vietnam and the Philippines.
All eyes will be on Trump's message to the North and Kim. Trump is expected to deliver a speech at the South's parliament and to visit an US military base during a November 7-8 trip to Seoul.
Trump's recent remark that "only one thing will work" with the North fuelled concerns of a potential conflict on the divided peninsula where the 1950-53 Korean War had left millions dead.
But Mattis has repeatedly stressed a diplomatic solution to ease tension during his trip to Asia this week, saying Washington was "not rushing to war" and its goal was "not war."
Some Trump advisers have said US military options are limited when Pyongyang could launch an artillery barrage on the South's capital Seoul -- only about 50 kilometres from the border and home to 10 million people.
The North is estimated to have some 10,000 artillery pieces and at least 50 short-range missiles stationed along its heavily fortified border with the South.
The country has made significant progress in its atomic and missile technology under Kim, who took power after the death of his father and longtime ruler, Kim Jong-Il, in 2011.
Since then he has overseen four of the country's six nuclear tests and hailed its nuclear weapons as a "treasured sword" to protect itself from potential invasion by the "imperialist enemy" the US.
Growing nuclear threats by the North prompted calls by some Seoul lawmakers to deploy tactical US nuclear weapons in the South, but Song dismissed such a possibility.
"We believe that tactical nukes are not worth deploying to the Korean peninsula," Song said, adding Seoul was capable of responding to potential nuclear attacks by the North without its own atomic weapon.