The project is seen as emblematic of warming relations between New Delhi and Tokyo, who are seeking to combat China's growing regional influence.
The high-speed link, which will use Japanese technology to connect the main city of PM Modi's home state with the country's financial capital, is scheduled for completion by December 2023.
"I hope to enjoy the beauty of India through the windows of the bullet train with Modi on my side when I come to India in a few years," Mr Abe told a ceremony in Ahmedabad.
His visit comes just after a lengthy standoff between India and China in a strategically important area of the eastern Himalayas, across the border at Sikkim.
Local media reports said Japan was likely to take up the sale of ShinMaywa US-2 amphibious aircraft that India has been interested in for several years.
The aircraft is expected to boost India's maritime capabilities.
PM Modi welcomed Mr Abe at the airport with his traditional big hug before the Japanese premier and his wife changed into traditional Indian clothes.
Japan is providing 85 percent of the cost of the new train link in soft loans.