Relations have strengthened between Asia's second and third largest economies as Mr Abe and PM Narendra Modi seek to balance China's rise as the dominant Asian power. Both men are nationalists who enjoy a personal friendship.
The deal to build a high-speed train line between the financial hub of Mumbai and the city of Ahmedabad gives Japan an early lead over China, which is conducting feasibility studies for high speed trains on other parts of India's dilapidated rail network.
"This enterprise will launch a revolution in Indian railways and speed up India's journey into the future. It will become an engine of economic transformation in India," Modi said in a speech.
Japan has offered a "highly concessional loan" at an interest rate of 0.1 percent rate with repayment over 50 years and a moratorium for 15 years, Indian Foreign Secretary S Jaishankar told a news conference.
India will be buying a Japanese high-speed train system, effectively with an export credit of $12 billion.
Under defence deals announced on Saturday, the two sides will share technology, equipment and military information, but the long-awaited sale of Japanese aircraft in a deal worth about $1.1 billion was not concluded.
Similarly, while they agreed to work towards cooperation in civil-nuclear technology, they stopped short of signing an agreement, citing outstanding technical and legal differences.
Mr Jaishankar did not cite a timeline for signing the final agreement with Japan.
Japan, the only country to have suffered a nuclear attack, has been demanding additional non-proliferation guarantees from India before it exports nuclear reactors.
India and Japan have been negotiating a nuclear energy deal since Japan's ally, the United States, opened the way for nuclear commerce with India despite its atomic bomb programme and shunning of the global Non-Proliferation Treaty (NPT).
A final deal with Japan would also benefit U.S. firms. India has already given land for nuclear plants to GE-Hitachi - which is an alliance between the U.S. and Japanese firms - and to Toshiba's Westinghouse Electric Company.
"The memorandum we signed on civil nuclear energy cooperation is more than just an agreement for commerce and clean energy, it is a shining symbol of a new level of mutual confidence and strategic partnership in the cause of peaceful and secure world," PM Modi said.
"I know the significance of this decision for Japan and I assure you that India deeply respects that decision and will honour our shared commitment," PM Modi added.
In a joint statement the two prime ministers mentioned the South China Sea and "called upon all states to avoid unilateral actions that could lead to tensions in the region".
China claims most of the South China Sea, through which more than $5 trillion in global trade passes every year. Vietnam, Malaysia, Brunei, the Philippines and Taiwan have rival claims.
New Delhi and Tokyo, both of which have territorial disputes with Beijing, have no claims in the waterway but worry about China's growing military reach into sea lanes through which much of Japan's shipborne trade passes. Abe and Modi called for freedom of navigation in international waters.
India and Japan have been holding talks for two years on the purchase by India of US-2 amphibious aircraft made by ShinMaywa Industries, which would be one of Japan's first arms sales since Abe lifted a 50-year ban on weapon exports.
Mr Jaishankar said a purchase of US-2 was discussed in Saturday's meeting and the "matter remains under consideration".
Get Breaking news, live coverage, and Latest News from India and around the world on NDTV.com. Catch all the Live TV action on NDTV 24x7 and NDTV India. Like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.