Herman Nackaerts, deputy director general of the International Atomic Energy Agency (IAEA), said on Friday his team had not been granted the access they have long sought to a military site. A further meeting was scheduled for February 12.
In a separate note sent to IAEA member states about the negotiations over Wednesday and Thursday, seen by Reuters, the UN agency said "important differences" between the two sides remained and it had therefore not been possible to reach an agreement.
The absence of an accord meant to allay international concerns over Tehran's atomic ambitions will disappoint world powers seeking a broader diplomatic settlement with Iran that would avert the threat of a new Middle East war.
The IAEA's efforts to unblock its long-stalled investigation into suspected atom bomb research in Iran are separate, but closely linked, to negotiations between Tehran and the powers that may resume later this month after a seven-month hiatus.
The IAEA, whose mission is to prevent the spread of nuclear weapons, has been trying for a year to negotiate a so-called structured approach with Iran giving the inspectors access to sites, officials and documents for their long-stalled inquiry.
At the centre of its concerns, it wants to inspect the Parchin military complex southeast of Tehran where it believes explosives tests relevant for nuclear weapons development may have taken place, something Iran denies.
"We had two days of intensive discussions," Nackaerts told reporters at Vienna airport.
"We could not finalise the structured approach to resolve the outstanding issues regarding possible military dimensions of Iran's nuclear programme," he said.
Nackaerts said as he left Vienna for Tehran on Tuesday that he hoped for immediate access to Parchin, where Western diplomats suspect Iran has carried out work to cleanse the site of any traces of past, illicit nuclear-related activity.
Iran's 'bargaining chip'?
Iran says Parchin is a conventional military facility and has dismissed accusations of "sanitisation" taking place there.
"Also on this occasion no access was granted to Parchin," said Nackaerts, who led a team of eight senior IAEA officials.
World powers were monitoring the IAEA-Iran talks for any signs as to whether Tehran, facing intensifying sanctions pressure, may be prepared to finally start tackling mounting international concerns about its nuclear programme.
Israel - a US ally believed to have the Middle East's only nuclear arsenal - has threatened military action if diplomacy and economic sanctions intended to rein in Iran's uranium enrichment programme do not resolve the stand-off.
Nuclear proliferation expert Mark Fitzpatrick said about the latest round of IAEA-Iran talks: "Unfortunately, this disappointing result is the norm. Despite all the optimism about engagement, in practice anyone who sits down at the table with Iran finds it immensely frustrating."
Diplomats have said there is still a window of opportunity for world powers to make a renewed diplomatic push to find an overall negotiated solution to the dispute after US President Barack Obama won re-election in November.
The powers - the United States, Britain, France, Germany, Russia and China - want Iran to scale back its uranium enrichment programme and cooperate fully with the IAEA. Iran wants the West to first lift sanctions hurting its economy.
Both sides say they want to restart their talks as soon as possible, perhaps later in January, but no date and venue has yet been announced.
"Iran may have an incentive to delay cooperation with the IAEA in order to use that as a bargaining chip in the more important political talks," Shashank Joshi, a Middle East specialist at the Royal United Services Institute, said.