He reached the holy Sikh shrine shortly after 6 AM and spent around 15 minutes inside its sanctum sanctorum.
Wearing a grey turban and a white shirt, Mr Sajjan took a round of the Harmandir Sahib's complex and spent over an hour in its premises.
Shiromani Gurdwara Parbandhak Committee (SGPC) President Kripal Singh was also present when the minister reached the Golden Temple. SGPC is the prime Sikh body involved in management of gurudwaras.
Mr Sajjan is a Sikh whose native village, Bambeli, falls in Hoshiarpur district of Punjab.
The Canadian defence minister reached the holy city of Amritsar from New Delhi last evening.
The minister will be in Chandigarh tomorrow where he will inaugurate the new office of the Consulate-General of Canada.
Former lieutenant Colonel in Canadian Army also visited Amar Jawan Jyoti at New Delhii on April 18.
Mr Sajjan also met India's Finance Minister Arun Jaitley to talk on trade and security. India has expressed strong objection to the recently passed motion by Canada's Ontario Assembly describing 1984 riots as 'genocide'. India has termed the resolution as 'a total exaggeration of facts'.