The man was detained on The Mall, the street that leads up to the Palace, after Metropolitan (Met) Police officers on patrol in the area stopped and searched him.
"Patrolling officers detained and searched a man in The Mall. A knife was recovered and the man was arrested on suspicion of a possession of an offensive weapon. He has been taken into custody at a central London police station. The incident is not believed to be terrorism related," a Met Police statement said.
The arrest came as the British capital went on high alert, with many of its prominent landmarks - including the Palace, Downing Street and Houses of Parliament - under additional security in the wake of the suicide bombing at an Ariana Grande concert in Manchester on Monday night.
The Met Police said it has increased police numbers and operations across London with immediate effect as Army troops arrived in the city as part of Operation Temperer, triggered last night when British Prime Minister Theresa May raised the country's threat level from "severe" to "critical".
Met Police Commander Jane Connors, leading the London policing operation, said: "The reality is that we must be prepared to be able to respond to and deal with armed and deadly attackers, so we must be in a position to respond with firearms officers who will use force to stop those attackers in their aim.
"That is why is we have increased the number of firearms who are on duty, both out walking and in roaming patrols, at fixed points and carrying out a range of operations."
"We are using every tactical option - not just through the use of armed officers but ongoing work by the Counter Terrorism Command; working with partner agencies and gathering community intelligence."
The force said that as a highly visible deterrent and disruptive tactic officers will be making more use of stop and search; vehicle check points and automatic number plate recognition (ANPR) technology.
Operations for upcoming events are also under review and there will be additional security arrangements put in place.
With the UK's threat level at its highest, 984 military personnel have already been deployed across the country.
As many as 3,800 will be available to back up police operations in following waves of deployment, UK Home Secretary Amber Rudd said.
In London, the Changing the Guard at the Palace was cancelled and Parliament was closed to its regular public tours.
The public galleries at the Old Bailey criminal court in London were also shut down.
Troops are likely to be on duty at this weekend's FA Cup final at Wembley stadium and the Twickenham rugby premiership final in London.
According to reports, the suicide bomber - Libyan-origin Salman Abedi - had travelled from London to Manchester by train just before blowing himself up, killing 22 people and injuring 119 others.
It has led to fears he may have had bomb-making links in the British capital.