Eyewitnesses said the pilot, Sumeeta Vijayan, crashed while trying to land in an open space.
But the former Air Force pilot failed, as the tail rotor of the helicopter got damaged and the aircraft got out of control. The single-engine chopper was built in 2010.
"A bird was found dead near crashed helicopter. It seems the helicopter collided with bird that led to crash. We'll be inquiring further," said Jammu and Kashmir deputy chief minister Nirmal Singh.
The helicopter belonged to the privately-run Himalayan Heli Services, which also runs choppers for the Chardham Yatra.
Many of them, especially the elderly people, use the chopper service, since the cave can be accessed only after a 10-km trek.
The bodies have been recovered and identified, sources said. Three of the deceased pilgrims were from Delhi and two from Jammu.
Ms Vijayan, who had been with the company for a decade and was its chief pilot, was from Gurgaon, Haryana. Her sister is also with the Air Force, and she lived in Jammu with her mother.
Thousands visit Vaishno Devi, one of the major pilgrim centres of India, every day. During the recent Navratras, more than one lakh people had paid obeisance at the cave shrine.
"It is a very unfortunate incident. We have formed an inquiry committee," said Ajit Kumar Sahu, the Chief Executive of the Vaishno Devi Shrine Board. The deceased will get Rs 25 lakh insurance cover and the shrine board will give Rs 3 lakh as compensation.