Mr Kagan was speaking to reporters after meeting Chief Minister Manohar Parrikar in Panaji.
"Because of the distance and travel time, marketing Goa as a stand-alone tourism destination is not possible.
"If you want a beach vacation, there are many destinations which are closer to the US," Mr Kagan said, responding to a question on promoting Goa as a tourism destination for the US citizens.
"Goa is a world-class tourist destination with incredible beaches, incredible culture but that is also a place that gives an access to so many incredible destinations in India," he said.
Over five lakh foreign tourists visit Goa's beaches every year, with those from Russia and the UK topping the list. On his meeting with Mr Parrikar, Mr Kagan said they had "very productive conversation".
"We talked about the relationship between US and India. As a minister of defence he did make serious efforts to strengthen our relationship," Mr Kagan said.
"There is tremendous opportunity in Goa and we are happy with the existence of number of American industries here, we are happy that Goa is a very famous tourist destination in the US. And we want to see that it increases," he said.