Holding that the petition against Mr Gandhi was not maintainable, a bench of Acting Chief Justice Gita Mittal and Justice C Hari Shankar said "we are not the appropriate forum" to decide on security aspects.
"We are not going to decide on security. For security also, we depend on the government. We rely on their assessment," the Acting Chief Justice observed and dismissed the plea.
It also said in case any action was required, the authorities were open to deal with it.
Centre's standing counsel Anil Soni said the government was also concerned about Mr Gandhi's security, but skipping it was "irresponsible" behaviour on his part. He submitted that if something happened to him, "we (government) will be held responsible."
The court was hearing a plea by Mumbai BJP spokesperson Tuhin A Sinha who had sought a direction to Mr Gandhi and the centre to ensure that the Congress leader does not violate the Special Protection Group Act and put himself in danger by giving a slip to the Special Protection Group.
The petitioner also sought a direction to Mr Gandhi to submit an affidavit to the court that he would not travel without taking SPG cover.
Mr Gandhi's car had come under attack from some stone-pelters when he was on a tour of flood-affected areas in Gujarat in early August.
Mr Sinha also reportedly said that Mr Gandhi had not followed the security protocol even when he travelled abroad.