General Constantino Chiwenga, 61, took the oath of office in Harare, pledging to be "faithful" to Zimbabwe and to "obey, uphold and defend the constitution".
"I will discharge my duties with all my strength and to the best of my knowledge and ability," said Chiwenga, dressed in a black suit at a ceremony held on the lawns of the presidential residence.
The new president Emmerson Mnangagwa, dozens of government officials, military and police chiefs as well as traditional leaders, attended the event.
Mnangagwa, who had a few weeks earlier been humiliated and sacked from his job as vice president by Mugabe, then took over as the head of state.
Mugabe, 93, was ousted from power after the military stepped in following internal feuding and factionalism that escalated in the ruling ZANU-PF party over who would succeed him. Mugabe's wife Grace had expressed an interest in succeeding her husband.
Chiwenga's ascent to the country's second most powerful job has further consolidated the military's power in the political space of the southern African country.
Several other senior army officers have in recent weeks been appointed to ministerial or influential party positions.
President Mnangagwa did not give a speech at the inauguration of his deputies, but told journalists that their responsibility is "to drive the ministers."
"The performance of the ministers will be reflected by the supervision they give."
Chiwenga became the face the of the rapid transition in Zimbabwe.
Surrounded by defence chiefs, Chiwenga called on the ruling party officials to "stop reckless utterances... denigrating the military".
The next day tanks rolled out of the barracks onto the streets of the capital.
Just before dawn on the following day, the military went on state television saying they were not launching a coup but were "targeting criminals around" Mugabe.
Chiwenga and Kembo Mohadi, a veteran politician and long-serving state security minister, were then named the ruling party's vice presidents.
Mohadi was also sworn in as the state second vice president on Thursday.
The appointment of Chiwenga and several other senior army officers in government and the ruling party is seen as a reward for the army's instrumental role in ending Mugabe's rule.
Two other top military officials were also awarded ministerial posts earlier this month.
Ex-air force chief Perrance Shiri became the new lands and agriculture minister and the general who announced the military takeover, Sibusiso Moyo, is the new foreign affairs minister.
Another senior military officer Lieutenant General Engelbert Rugeje was early this month appointed ZANU-PF national political commissar, a powerful position equivalent to organising secretary.