The assailant "died on the spot" after the guard opened fire, police spokesman Harald Soeros told AFP, adding that the attack took place just before midnight local time in a residential area.
The motive for the attack is not yet known, police said.
The guard, who was in a sentry box outside the building, suffered a wound to his upper arm in the attack.
"He initially used pepper spray" against the assailant, Soeros said, before opening fire.
According to early indications the guard "acted within the rules", said a spokesman from the ministry of defence, who said the attacker lived in Vienna but originated from the Tyrol region of Austria.
Police have ordered reinforcements around all diplomatic missions in the city.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)