Cuomo said he learned on Friday of "alarming" levels of radioactivity at three monitoring wells at the Indian Point plant in Buchanan, about 40 miles (65 km) north of New York City on the east bank of the Hudson River.
The governor has previously asked the Nuclear Regulatory Commission to shut down the plant because of doubts over the safe evacuation of the area in the event of an accident.
Cuomo said in a statement the plant's operator, Entergy Corp, has informed him the contaminated water has not migrated off the site and poses no public health risk.
The governor said he has directed the state departments for the environment and health to investigate the leak.
"Our first concern is for the health and safety of the residents close to the facility and ensuring the groundwater leak does not pose a threat," Cuomo said.
Entergy in a statement said the tritium likely reached the ground at Indian Point during recent work at the site.
"While elevated tritium in the ground onsite is not in accordance with our standards, there is no health or safety consequence to the public, and releases are more than a thousand times below federal permissible limits," the company said.
Cuomo, in his letter ordering the state investigation, said Indian Point has previously had a problem with the release of radioactive water, and this time the levels of radioactivity reported by the company are worse than in past incidents.
One monitoring well showed a nearly 65,000 per cent spike in radioactivity, from 12,300 picocuries per litre to more than 8 million picocuries per litre, the governor said.
On May 9, 2015, a unit at the Indian Point nuclear plant was shut down following a transformer failure and fire, in an incident that sent smoke spewing from the plant.
But the plant remained stable and the NRC that day said there was no threat to the public.
The company is seeking to relicense the two reactors at Indian Point for another 20 years, but Cuomo has asked the NRC to deny Entergy's application for relicensing.