Mr Herzog was a member of Chancellor Angela Merkel's conservative Christian Democratic Union. He served as the chief justice of Germany's highest court before being elected to a five-year term as President in 1994, four years after reunification.
Germany's current president, Joachim Gauck, described Mr Herzog as a "distinctive personality, who had helped shape Germany's self-image and the interactions in our society."
Mr Herzog last year hit out at the big political parties for not doing more to halt the rise of the anti-immigrant Alternative for Germany (AfD) party, which looks poised to become the third largest party in parliament in national elections this year.
After his term as President, a largely ceremonial post in Germany, Mr Herzog also chaired the first European convention that drafted the European Union's Charter of Fundamental Rights.
"Today we have lost a great constitutional scholar, politician and statesman," said German Foreign Minister Frank-Walter Steinmeier in a statement.
He described Mr Herzog as "a straightforward, honest and smart person who was not afraid to speak difficult truths, but also never lost his profound sense of humour."