The project is seen as emblematic of warming relations between New Delhi and Tokyo, who are seeking to combat China's growing regional influence.
The high-speed link, which will use Japanese technology to connect the main city of PM Modi's home state with the country's financial capital, is scheduled for completion by December 2023.
"I hope to enjoy the beauty of India through the windows of the bullet train with Modi on my side when I come to India in a few years," Mr Abe told a ceremony in Ahmedabad.
His visit comes just after a lengthy standoff between India and China in a strategically important area of the eastern Himalayas, across the border at Sikkim.
The aircraft is expected to boost India's maritime capabilities.
PM Modi welcomed Mr Abe at the airport with his traditional big hug before the Japanese premier and his wife changed into traditional Indian clothes.
Japan is providing 85 percent of the cost of the new train link in soft loans.
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.