WASHINGTON (AP) _ Supporters and opponents of President Reagan's $100 million aid package to Nicaraguan rebels tried and failed today to fashion a compromise that would win widespread Senate support.

Senate Majority Leader Bob Dole, R-Kan., told his colleagues shortly after noon that ''we were unable to agree'' on a compromise, leading to the prospect of a number of amendments being offered.

The Senate was scheduled today to begin up to 10 hours of debate on the aid plan. But by late afternoon, the issue still wasn't formally on the floor, meaning a vote was not likely to occur until Thursday.

After the compromise effort failed, Democratic opponents of the Reagan aid plan met in a lengthy closed-door session to discuss strategy in the debate over the aid proposal. The meeting delayed the formal opening of the debate.

Dole said backers of Reagan's plan believe they have the votes to win outright, but tried to work out a compromise to pick up as many Democratic votes as possible.

''If we can get a large vote here,'' Dole said, ''we can send a clear, strong message to Managua'' that the American government is united.

Dole said the reported incursion by Nicaraguan troops into neighboring Honduras improves chances for Senate approval of the president's plan.

Dole, referring to Nicaraguan President Daniel Ortega, told reporters that ''Ortega gave us a boost'' by moving troops into Honduras. ''I hope they don't pull out today.''

Earlier, Dole and Sen. Richard Lugar, R-Ind., chairman of the Foreign Relations Committee, had met with Sen. James Sasser, D-Tenn., chief proponent of a Democratic alternative to Reagan's plan, to try to fashion a compromise.

Sasser said ''we're trying to work something out'' as he entered the meeting with Dole and Lugar.

Congressional sources said the compromise effort had failed essentially over one point: the mechanism by which military aid for the Contras would be approved after a 90-day period in which the United States sought to open negotiations with the Sandinistas.

Senate Democratic Leader Robert C. Byrd, D-W.Va., and Adm. John M. Poindexter, President Reagan's national security adviser, reportedly failed to agree on that point after Poindexter joined the meeting in Dole's office.

Byrd had sought an affirmative vote to approve the military aid after the expiration of the 90-day negotiation period, a situation in which either chamber could block it.

The administration held to its position that the aid could be stopped after the 90-day period only by the negative votes of both houses, a situation which would require two-thirds majorities in the House and Senate to overide a certain presidential veto.

The failure to strike a compromise left the final outcome less clear, opening the way for votes on several possible amendments which would offer a variety of alternatives to the president's aid plan, including the option of killing it outright.

The Senate was still expected to approve some form of assistance to the Contras, however.

Sen. John Warner, R-Va., a supporter of Reagan's proposal, said Tuesday that reports that as many as 1,500 Nicaraguan troops had crossed into Honduras to attack Nicaraguan Contras was ''going to have a dramatic impact in the Senate.'' He predicted ''it certainly lessens the need for any compromise'' with Reagan's opponents.

Nicaragua's Sandinista government denied it had invaded Honduras. However, Reagan on Tuesday provided $20 million in emergency military aid to Honduras, which confirmed the Nicaraguan invasion. Reagan also said U.S. helicopters and pilots will provide transportation assistance to the Hondurans, according to White House Chief of Staff Donald Regan.

News of the incursion angered congressional Democrats opposing Reagan's plan. House Speaker Thomas P. O'Neill Jr., D-Mass, who led the opposition last week when the House rejected Reagan's proposal, called the incursion ''a tremendous blunder on the part of Ortega.''

He said of Ortega, ''I have never met him, but I think he's a bumbling, incompetent, Marxist-Leninist communist.''

O'Neill was scheduled to leave today on a trip to Latin America, leading a bipartisan group of House members on a visit to nations he said hav demonstrated support for a regional, diplomatic solution to the conflict.

The Democratic-controlled House voted 222-210 last week against Reagan's aid package, which includes $70 million in military assistance and $30 million in non-lethal help such as clothing and medicine.

The aid would go to the Contra guerrillas fighting Nicaragua's leftist government, chiefly from bases in neighboring Honduras. A separate program of $27 million in non-lethal aid expires next Monday.

Senate opponents of Contra aid have tried to come up with a compromise package that would provide some aid but also have several restrictions.

Sasser has led Democratic efforts to draft a compromise which would provide non-lethal aid now and military aid after six months, but only if Nicaragua had refused to negotiate to end the impasse.

Lugar called the Nicaraguan incursion ''a very important factor in changing the atmosphere'' in Congress.

''I think everybody knows now that Nicaragua invaded Honduras with quite a force,'' he said.

Dole said Ortega ''shot himself in the foot'' and told reporters GOP senators now believe Reagan's package will win without compromise.

Sen. Patrick Leahy, D-Vt., an opponent of the Reagan aid package, said, ''I've heard the rumor - perhaps it's tongue in cheek - that Daniel Ortega is secretly on the payroll of one of our intelligence agencies and is a lobbyist for President Reagan's positions.''

Regan, after meeting with Republican senators at the Capitol, told reporters the administration does not see the need for a compromise.

He said the administration is willing to talk to the Sandinistas, but only if Ortega's government conducts separate negotiations with the Contras, a step Nicaragua has refused to take.

The Contra aid debate had been scheduled to begin Tuesday, but many Republicans wanted it delayed until today to permit senators time to let the implications of the Nicaraguan action ''sink in,'' Lugar said.

Any aid package approved by the Senate will eventually be sent back to the House, where O'Neill has promised it will be considered April 15.