The Taiwanese man who was arrested at the scene said he was expressing his political views and had stolen the sword from a nearby history museum, police told AFP.
The presidential office in the centre of the capital Taipei is the headquarters of Taiwan's Beijing-sceptic President Tsai Ing-wen.
Relations with Chinese authorities have deteriorated since she took office last year as she has refused to agree to Beijing's stance that Taiwan is part of "one China".
The island is a self-ruling democracy, but Beijing still sees it as part of its territory to be reunited.
The attacker "took a hammer and smashed a display case in a history museum to steal a samurai sword", a police official working on the incident, who did not want to be named, told AFP.
"A Chinese national flag was found in his backpack. He said he wanted to express his political stance by going to the presidential office," the official said.
The man, identified only by his family name Lu, stabbed the officer as he tried to stop him entering the complex from a side gate, said presidential spokesman Alex Huang.
Mr Lu is currently being questioned by police.
Local media said the guard had been stabbed in the neck and that the presidential office was hosting a family event for its staff at the time.
TV footage showed Mr Lu being carried away by four officers and put inside a police car at a side entrance to the presidential office, which has been cordoned off since the attack.
The complex and its surroundings have been the target of attacks before.
In November 2014, a driver tried to smash his vehicle into the front door of the nearby presidential residence, saying he was protesting the health policies of Tsai's predecessor Mr Ma Ying-jeou, who was in power at the time.
In January 2014, a man drove his truck through a bullet-proof screen and into the main gate of the presidential office, saying he was protesting over a judicial hearing involving his ex-wife.
There were no injuries from either incident, apart from one of the drivers who was wounded.