According to fire department officials, at least four explosions were heard at the Chidambaram Fireworks factory in the Vilampatty village. Seventeen of its 40 units were gutted in the fire.
The authorities said that the explosions took place during lunch hour when most of the workers were away, preventing casualties and injuries.
Fire tenders have been rushed to the spot. The police said that the fire spread to an adjacent factory as well.
Sivakasi is over 500 kilometres south of Chennai. It is known for its cracker and match factories and according to estimates, the city manufactures 70 per cent of India's total firecracker produce.
With Diwali just a few months away production at cracker factories would be at its peak.
Last September, 38 workers lost their lives in a fire accident at a factory near Sivakasi.
In April this year, a similar explosion at a manufacturing unit in the area killed eight people. Three people were arrested in the case.
The victims were workers of a unit at Narayanapuram village, four kilometres from Sivakasi and were engaged in making crackers when the explosion occurred.
The explosion apparently occurred due to friction between the chemicals.