Although Mr Goel withdrew the pictures -- taken in a restricted zone -- today, some ministers from the opposition parties demanded his resignation over the controversy.
The pictures were taken last year when Mr Goel visited the temple and performed rituals inside the "Garva Griha" of the 11th-century shrine.
He posted the pictures on Sunday on Twitter while wishing the people of Odisha on the occasion of Utkala Dibasa.
In his Twitter post, Mr Goel wrote, "Greetings of #UtkalaDibasa to our brothers, sisters and friends from #Odisha! Some pictures of my trip to the majestic Lingaraja Temple in Bhubaneswar last year."
However, the pictures drew criticism from various quarters following which Mr Goel took down the image on Monday.
Mr Goel said he had tweeted the pictures "by mistake" and did not wish to hurt anyone's sentiments.
"On the occasion of #UtkalaDibasa, I had by mistake tweeted pictures of the Lingaraja Temple from my trip last year. I did not wish to hurt anyone's sentiments and therefore, I am withdrawing the tweet/images," Mr Goel said in another tweet today.
"Either the union minister should resign or be driven out of the Union Cabinet by the prime minister. He should tender unconditional apology to the people of Odisha or else there would be a statewide agitation," Mr Routray said.
BJD leader and Rajya Sabha member Prasant Nanda said the photo was "against the religious tradition".
"The minister should tender unconditional apology to the people of Odisha the way Bollywood actress Rabina Tandon had done after she had a shooting session inside the Sri Lingaraj temple in March this year," Mr Nanda said.
BJP Odisha state general secretary Prithviraj Harichandan came to Mr Goel's defence, saying the union minister wasn't briefed by either the temple administration or the temple servitors about the ban on photography inside the temple.
"If this action has hurt sentiments of the people, we tender apology," Mr Harichandan said.
The Executive Officer of Sri Lingaraj Temple, Rajiv Parida, termed the incident as "unfortunate".