Hanako, which means "Flower Child" in Japanese, became something of a cause celebre last year following an international campaign to improve the ageing pachyderm's cramped living conditions.
The cause of death was not immediately known, zookeepers told AFP, adding that an autopsy would be conducted later in the day. Captive elephants have a life expectancy of 40-plus years.
According to Guinness World Records, the oldest known elephant was Lin Wang, an Asian elephant who lived until the grand old age of 86 and died in 2003 at Taipei Zoo.
Mourners flocked to Tokyo's Inokashira Park Zoo to pay their respects on Friday with more than 70 condolence cards left for Hanako by well-wishers.
"Fans are visiting the park to place flowers in front of Hanako's enclosure," said Hiroshi Mashima, in charge of information and education at the zoo.
Hanako passed away on Thursday after 20 zoo staff members attempted to raise her to her feet by rope, a common technique used when elephants remain lying on the floor, according to Mashima.
Elephants die if they lie on their side for a prolonged period of time as it can crush their internal organs, Mashima added.
"She passed away quietly and calmly," Kiyoshi Nagai, head of the zoo, was quoted as saying by Japan's Kyodo news agency.
"It's truly a pity. She was the most beloved elephant in Japan."
Hanako, who lived longer than the average 55-60-year life span of wild elephants, became a media star last year after a heart-wrenching blog post by a Canadian animal rights activist led to an online petition.
"I was shocked and dismayed to see the conditions of her confinement first-hand," wrote Ulara Nakagawa.
Follow NDTV for latest election news and live coverage of assembly elections 2019 in Maharashtra and Haryana.
Subscribe to our YouTube channel, like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.