"After negotiations, the two parties (Israel and the family) have reached an agreement whereby the state will pay 4 million shekels ($1.1 million or 842,000 euros) to the deceased's family," the ministry said in a statement late on Tuesday.
The family of Ben Zygier, an Australian-Israeli known as "Prisoner X," had accused Israel of negligence in dealing with his case, according to the statement.
Zygier was found hanged in his isolation cell in Ayalon prison near Tel Aviv in December 2010 -- a case Israel went to extreme lengths to cover up.
A court document released on April 25 this year said Israel's prison service had caused Zygier's death by failing to prevent him from committing suicide.
The document revealed details about his background and imprisonment, indicating he was suicidal and had an emotionally-charged exchange with his wife the day he was found hanged.
It also said that his cell was not properly watched by prison guards.
The justice ministry statement stressed that the deal with Zygier's family was not an "admission of alleged wrongdoing".
It was instead "to avoid the affair going to court, which would lead to the publication of numerous details of the case which could cause serious harm to national security."
The reasons for Zygier's detention were unclear, but the Australian Broadcasting Corporation said in a report in May that the 34-year-old, who was allegedly working for Israel's foreign spy service Mossad, had unwittingly sabotaged a top secret spy operation in Lebanon.