Police said the fire broke out at 4 pm in a wooden cooling tower at the state-owned oil refinery, where private workers had been brought in for welding work. A thick layer of smoke engulfed the area causing tension and panic among nearby residents.
The fire has now been brought under control. Police say casualties could rise.
A fire had reportedly broken out in the same plant earlier this year, but had been put out before any major disaster.
The Visakhapatnam plant is one of the two Hindustan Petroleum refineries in India, the other being in Mumbai.