Dr Kalam, who as a scientist played a leading role in the country's landmark 1998 nuclear weapons tests, served as president from 2002-2007 and a strongly secular figure.
The sculpted "Bhagvad Gita" was placed next to the statue before it was inaugurated by Prime Minister Narendra Modi last week on the second anniversary of Dr Kalam's death.
Relatives placed the Bible and the Quran next to the Gita in protest. That infuriated the Hindu Makka Katchi party, which said it made the complaint to police.
"The authorities removed the Bible and the Quran at the site. We are also looking at the complaint filed with us," police superintendent Om Prakash Meena told AFP.
Authorities have since prevented photographers from taking images of the statue.
Memorial officials did not respond to queries.
Dr Kalam was born into poverty and sold newspapers as a child to support his family while studying. His work on India's own nuclear weapon earned him the nickname of "India's missile man".