PARIS (AP) _ The Soviet Union and other nations said Thursday they will give up about $6 million in refunds from a special UNESCO fund to ease the financial loss caused by the pullout of the United States, which had provided one-quarter of the agency's budget.

France, UNESCO's host country, announced Wednesday that it would make a special payment of $2 million. The agency lost an annual contribution of $43 million when the United States withdrew at the end of last year.

Director-General Amadou Mahtar M'Bow said UNESCO must make up a $28 million deficit because of the American withdrawal.

Indian delegate T.N. Kaul said Thursday that the U.S. action, and threats by some other members to follow Washington's lead, put the entire U.N. system into serious jeopardy.

Critic nations say the agency is mismanaged, politicized and fiscally extravagant.

''We are faced with a situation that must change for the better or destroy the twin pillars of international cooperation on which UNESCO in particular and the U.N. system in general is founded,'' Kaul said.

He told a meeting of the U.N. Educational, Scientifc and Cultural Organization's executive board that there is no alternative to UNESCO and the United Nations ''in the interdependent thermonuclear world of today.''

''Let us not destroy the system we have built up for the last 40 years and try to rebuild it on the ashes of what may survive,'' he said.

The board was meeting to consider the consequences of the U.S. pullout. Kaul, a former ambassador to both Washington and Moscow, was elected chairman of a group that will make recommendations on UNESCO's future course.

He expressed regret at the U.S. departure, decisions by Britain and Singapore to leave at the end of 1985, and Japan's announcement Wednesday that it will reconsider its membership.

The Soviet Union and several countries in Latin America, Africa and Asia said they would help meet the cash shortage by foregoing refunds totaling about $6 million due them from a currency fluctuation fund.

The biggest contribution, about $2.5 million, came from the Soviet Union. Mexico said it would give up its refund of $500,000 and India said it was considering the sacrifice of a similar amount.

Some Western officials said privately they were disappointed at France's special contribution because it was contrary to sentiment in their group that action should be taken to cut UNESCO's 187 sub-programs.

Several speakers said spending reductions must be made no matter how much money is raised by special appeal, and called on M'Bow to start by cutting administrative and staff costs.

Mongolia and the Ukraine said money could be saved by reducing from two to one the number of yearly meetings of a UNESCO commission that deals with human rights violations.

Several Western nations said they favored selective cuts in UNESCO's programs, but many African countries opted for an across-the-board reduction by a specific percentage for each program.

There also was support from several speakers for allowing the United States to have an observer mission, but the Soviet Union said there was no point in discussing the matter without a formal request.

The Americans informed UNESCO by letter Jan. 10 that they had a mission ready, but M'Bow said a report to the board meeting that there was no provision in the constitution for such a delegation.