KHARTOUM, Sudan (AP) _ Security men foiled an attempt to topple the fragile coalition government of Prime Minister Sadek Mahdi and arrested more than two dozen members of a previous administration, officials said.

Defense Minister Abdul-Maguid Khalil said the coup was attempted on Saturday against the Moslem government, whose factions are quarreling over a proposed peace agreement with Christian and animist rebels.

He confirmed a report in the newspaper Al-Siyassa, which said security personnel arrested 25 retired military officers and former civilian officials from the former government of President Gaafar Nimeiri.

Nimeiri was ousted in a coup April 6, 1985.

The Al-Siyassa report was the third newspaper report of an attempted coup in Sudan in the past two weeks.

After the report appeared, about 300 demonstrators ignored an emergency government order, gathering in Khartoum's Liberty Square to demand government approval of a tentative agreement to stop Sudan's civil war.

Relief officials estimate at least 500,000 people died this year from famine due to drought and civil war.

Riot police and plainclothes security men ringed the square as protestors chanted ''No, no to war'' and ''Yes, yes to peace.''

The Democratic Union Party, the second-largest member of the government coalition led by Mahdi's Umma Party, signed the peace agreement with the Sudan People's Liberation Army on Nov. 16.

The tentative pact calls for a cease-fire, a conference to negotiate a permanent peace, a new constitution and a freeze on implementing Sudan's Islam-based legal code.

Rebel leader John Garang began his civil rebellion in early 1983 to press demands for what he considers fairer economic, political and administrative treatment for southern Sudan.

The crowd broke up only after they were asked to leave by Abbas Awad, a member of the organizing committee representing 54 political parties and labor groups supporting the peace agreement.

''We have specific information about plots aimed at jeopardizing democracy in the country and that the demonstration of today would have been used as a springboard for this purpose,'' Abbas told the crowd.

Interviewed later, he said the Democratic Union Party canceled the demonstration because a ''group was planning to counter (it) . .. to spread chaos so that the army would intervene.''

Parliament voted 136-78 Wednesday to authorize Mahdi to convene a constitutional peace conference Dec. 31 and invite the rebels to attend, according to the Egyptian state-run Middle East News Agency.

The conference is a key clause of the peace agreement, but the Democratic Unionist Party and Garang have been insisting that the entire accord be approved before convening the conference.

The vote in effect overruled the Democratic Unionist Party demand, prompting some of its members to threaten to withdraw from the coalition that also includes the Moslem Fundamentalist National Islamic Front.

The front oppposes the peace plan because it calls for freezing Sudan's Islamic legal code until an alternative was reached.