Authorities later used equipment from their explosives unit to disarm the objects. They said they took the two objects away to be studied, to determine if they were actually dangerous.
"This will take some time," Berne's cantonal police said in a statement.
The 21-year-old man, who was arrested, was taken into custody for further questioning.
The area around the Heiliggeistkirche, or Church of the Holy Ghost, was temporarily closed for security reasons and transport re-routed, police said, later adding that all areas had been reopened.
(Reporting by Brenna Hughes Neghaiwi, editing by John Miller and Hugh Lawson)
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)