At least two of the four attackers have been killed by Afghan special forces. Ministry of External Affairs spokesperson Vikas Swarup said that no Indian casualties had been reported.
"Everybody is safe," B Sarkar, Consul General at the consulate said. The mission in the northern Afghan city is manned by three Indian personnel.
No group immediately claimed responsibility for the attack. Mr Sarkar said the shooting lasted for 20 minutes. "They shot from some adjacent building but nobody could enter the consulate."
Muneer Ahmad Farhad, a spokesman for the Balkh province governor, said the attackers had hidden in a house near the consulate and struck after darkness fell.
A local police spokesman in Mazar-i-Sharif, a city in the relatively tranquil province of Balkh, said security officials had cordoned off the area where sporadic gunshots were ringing out after a series of explosions.
The consulate assault is the latest in a series of attacks on Indian targets in Afghanistan.
Nine civilians, including seven children, were killed in August 2013 when suicide bombers targeted the Indian consulate in the main eastern Afghan city of Jalalabad, detonating an explosives-packed car.
The incident comes as Indian security forces are still trying to suppress an attack on the air base in Pathankot, near the border with Pakistan that has killed at least seven military personnel and wounded 20 others.
It also came amid renewed efforts by Prime Minister Narendra Modi and his Pakistani counterpart Nawaz Sharif to reduce longstanding tensions between the two countries.