NICOSIA, Cyprus (AP) _ Iran's foreign minister said today efforts to free Western hostages in Lebanon had reached a ''dead end'' because the hostage-takers had gotten nothing for captives released earlier, Tehran radio reported.

Repeating Iran's previous conditions for working for the release of the hostages, the official, Ali Akbar Velayati, said that hundreds of Arab prisoners must be freed by Israel and its proxy, the South Lebanon Army militia.

The comment comes amid several newspaper reports indicating that prospects were good for a release soon of the 13 Westerners held by pro-Iranian extremists in Lebanon.

Velayati said relations between Tehran and Washington, severed in 1979, could improve only if billions of dollars of Iranian assets frozen in the United States were unblocked.

But he did not link that issue to the hostages. Iran denies any involvement with the hostage-holders, saying it only has influence with them.

The radio, monitored in Cyprus, said Velayati was speaking in an interview with two U.S. television networks.

Dutch Foreign Minister Hans van den Broek, who returned from a visit to Tehran on Thursday, said that that the Iranian government was working to secure the release of Western hostages.

''Iran is ready and already making efforts'' to win freedom for the captives, he said.

Six Americans are among those missing and believed held hostage by pro- Iranian Shiite Muslim fundamentalist groups in Lebanon. The longest-held is Terry Anderson, chief Middle East correspondent for The Associated Press, who was kidnapped March 16, 1985.

Sources in Tehran had said that a deal for the release of the hostage was being worked out, and that the visit of Swiss Foreign Minister Rene Felber to Tehran last week had been significant.

The Swiss are used as intermediaries between Tehran and Washington.

When asked in the interviews when the Westerners could be freed, Velayati said that those efforts had ''reached dead end.''

Referring to the release last year of two American and one British hostage by their Lebanese captives, Velayati said that ''we were able to free some these hostages.

''But currently, because the Lebanese groups who freed the hostages gained nothing from the United States, and in fact have witnessed negative reactions from it, we have reached a dead end.''