The Tokyo High Court on Monday reversed the ruling dating from 2014 that granted Iwao Hakamada a fresh trial and saw him released from prison.
He was detained for 48 years, most of them in solitary confinement, and spent 45 of those years on death row following his conviction in the 1966 murder of his boss and the man's wife and two children.
"It's regrettable... we'll move to the next step," Hakamada's sister Hideko told supporters on Monday. Hakamada's defence team told local media they would appeal the decision to the Supreme Court.
Hakamada will not be jailed again pending the appeal and his death sentence continues to be suspended, given his age and his frail health which makes it unlikely he would flee.
Hakamada was believed to be the world's longest-serving death row inmate and his case attracted international attention.
He was freed in March 2014 when the Shizuoka District Court ordered a fresh trial on concerns that investigators could have planted evidence in order to clear up a crime that had shocked the country.
Prosecutors appealed that decision, resulting in Monday's ruling.
After his arrest in 1966, Hakamada initially denied accusations that he had robbed and killed his boss and his family before setting their house ablaze.
But the former professional boxer later confessed, following what he subsequently claimed was a brutal police interrogation that included beatings.
Prosecutors and courts used bloodstained clothes, which only emerged a year after the crime and his arrest, as key evidence in convicting Hakamada.
But his supporters said the clothes did not fit him and the bloodstains appeared too vivid for evidence that was discovered so long after the crime.
Later DNA tests found no link between Hakamada, the clothes and the bloodstains, his supporters and lawyers said.
His sister Hideko campaigned tirelessly for his release and has told AFP previously that Hakamada appeared to have developed mental illness after years of solitary confinement.
On Monday the high court questioned the DNA test submitted by Hakamada's lawyers. This had prompted the ruling for a new trial by showing no link between his DNA and the bloodstained clothes.
The court said Monday that the DNA analysis method was unorthodox and there was "serious doubt" about it, according to local media.
Japan is the only major industrialised democracy other than the United States to carry out capital punishment.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)