31 assault rifles and around 5,000 rounds of ammunition were recovered from the ship that was detained at the Tuticorin coast in Tamil Nadu early on Saturday morning, 600 km south of Chennai.
The MV Seaman Guard Ohio was not cleared to enter Indian waters. For three days, it remained undetected.
The ship is reportedly registered in Sierra Leone but is owned by an American company which specialises in maritime security against pirates. Representatives of the firm in India have met with Coast Guard officials to discuss the case.
A police case has been filed against the 35 people on board for illegally transporting arms and ammunition. They have all been detained and the captain has been questioned.
The ship had reportedly been in Indian waters for three days before it was detected. Local fishermen tipped off the Coast Guard that they were being approached by the Seaman Ohio's crew to buy supplies.
The Tamil Nadu government has shared a report based on the interrogation of the crew with the Intelligence Bureau and RAW. The case has now been transferred to the specialised wing of the Tamil Nadu, known as the Q Branch.
"You have had piracy earlier in the Strait of Malacca and now in the Gulf. People who provide such security need a floating armoury. My suspicion is that the boat in Tuticorin is a floating armoury for one of these private security agencies," said Deputy National Security Adviser Nehchal Sandhu.
Eight Indians were among the 35 people on board which included a 10-member crew and security guards.
The ship was also carrying 1,500 litres of diesel which had allegedly been bought illegally. Investigators are trying to locate two Indians who supplied the fuel.
Sources said this is the second time in the last two months that the ship has been detained by the Coast Guard. But the last time around, those who searched the ship said nothing suspicious was found on board.