During his brief visit, Mr Tillerson was given a tour of the room at the Birla House where Gandhi had spent his last 144 days.
Dressed in a grey suit, the US Secretary of State later paid tribute to Gandhi at the Martyr's Column, a monument erected at the spot in the lawns of the sprawling House, where he was killed on January 30, 1948 by Nathuram Godse, when he was taking a stroll there.
The House was converted into a memorial after his assassination. A museum, an art gallery, a library and bookstore are housed in its well-kept white bungalow.
A state of Mahatma sculpted by noted artist Ram Sutar welcomes visitors near the main entrance, in its premises which includes a manicured lawn at the back and a giant world peace gong in the front of the building.
Dipanker Shri Gyan, director of the memorial accompanied Mr Tillerson during his visit. "The US Secretary of State spent about 10 minutes at the memorial. He first visited the room in which Gandhiji had stayed before his assassination and was struck by its aura," he said.
"His immediate reaction after seeing the room was -- 'Such a simple man.' Gandhiji's life, indeed was simple and inspiring, two rare qualities," Gyan told PTI. Mr Tillerson then while walking to the Martyr's Column, stopped mid-way to enquire about the stone-made footsteps, which were covered with rose petals.
"He asked me about the footsteps, and I told him they symbolised the last steps taken by the Mahatma before he was shot dead. Tillerson then remarked -- 'Gandhi was one of the ten global leaders who changed the world.' He then paid homage to him at the Martyr's Column," Mr Gyan said.
Charge d'Affaires at the US Embassy here MaryKay Carlson and various other senior officials of the American mission were also present at the memorial.
Mr Tillerson was to visit the Gandhi memorial around 11 AM. His visit was, however, delayed a bit owing to his busy schedule, and he finally arrived around 11:20 AM.
The list of visitors to the memorial located on Tees January Marg, includes the who's who of the world. Duke and Duchess of Cambridge Prince William and Kate Middleton had visited it in April last year.
The royal couple had spent about 45 minutes there. They also saw the exhibits and came to know that Gandhi had learned to wear a tie and play violin during his stay in London as a student.
Lord Mountbatten, the last Viceroy of India was related to the British royal family. Gandhi and Mountbatten were known to have shared bonhomie and the last Viceroy even attended his funeral.
Gandhi had stayed at Birla House several times from 1939-1948, according to a plaque installed on a wall of the memorial.