AirAsia said the plane suffered a "technical issue" with Australian media reporting that the aircraft had to quickly drop from 32,000 feet to 10,000 feet 25 minutes after takeoff.
Video circulating online shows distressed passengers wearing oxygen masks with an alarm blaring and AirAsia staff calling for people to assume the brace position.
"I picked up my phone and sent a text message to my family, just hoping that they would get it," one tearful passenger named Leah told Channel Nine television. "It was horrible."
Another holidaymaker said not knowing what was going on heightened fears.
"We didn't know what was happening because all the voice recordings on the plane where in every language but English," she said.
AirAsia apologised for the scare, blaming a "technical issue" without elaborating on the cause.
"The safety of passengers and crew is our priority," the budget airline said in a statement.
"AirAsia apologises to passengers for any inconvenience caused."
Several flights have been forced back to Australia in recent months, including an AirAsia Gold Coast to Kuala Lumpur service in July that the carrier said involved a suspected bird strike.
A Qantas flight enroute to Dallas returned to Sydney in September after the wing flaps could not be retracted, while a Johannesburg-bound plane turned back to Sydney in the same week when a crack in the windscreen was discovered.