MANAGUA, Nicaragua (AP) _ A crewman of a Contra supply plane shot down by Sandinista troops said the aircraft took off from a CIA-manned base in Honduras, and the Nicaraguan government claimed the pilot worked for the CIA.

President Daniel Ortega said late Sunday that the statements by Alejandro Sanchez Herrera proved Honduras is violating provisions of a Central American peace plan by allowing anti-Sandinista rebels to operate out of Honduras.

The Defense Ministry said four crewmen - three Nicaraguans and a Colombian pilot - were killed when the vintage DC-6 supply plane was shot down with a Soviet-made, surface-to-air missile over southern Nicaragua late Saturday.

There were no Americans on board, according to the government and the U.S.-supplied Contras.

Sanchez Herrera, a Nicaraguan, said the pilot was known to the other crew members only as ''Richard.'' The Defense Ministry said the pilot worked for the CIA. Sanchez Herrera was allowed to talk to reporters Sunday while under military custody in the town of San Carlos, a few miles from the crash site.

Sanchez Herrera said two men parachuted from the plane with a supply of food and ammunition for the Contras before it was shot down. He didn't identify them but said he believed the men had been trained in the United States in the handling of explosives.

He told reporters the plane took off from a CIA-manned base in Swan Island in Honduras and that about 30 people from the CIA directed the mission from the island. Swan Island has been used for decades by the American military to monitor shipping and military activity in the area.

Spokesmen at the State Department were not immediately available Sunday night to comment on the charges against the CIA.

Addressing leaders of the National Farmers and Ranchers Union, Ortega said Sanchez Herrera's statement is proof that Honduras is allowing the United States to use its territory in activities against Nicaragua.

''President Reagan should reflect over this, since his policy is being shot down just like this plane was shot down ... Even though they may approve more funds (for the Contras), they will never be able to defeat our people,'' Ortega said, referring to Reagan's efforts to get Congress to approve more Contra aid.

''The (peace) accords continue to be violated, because they continue to use the territory of Central American countries - in this case Honduran territory - to sow death in Nicaragua,'' Ortega said.

The peace plan which Ortega and the presidents of Honduras, El Salvador, Guatemala and Costa Rica signed last August prohibits the signatory nations to allow insurgents of other countries to operate on their territory.

It also calls for cease-fires in the region's wars, democratic reforms and amnesty for political prisoners.

Earlier, both Ortega's brother, Defense Minister Humberto Ortega, and Contra spokeswoman Marta Sacaza in Miami had said there were 11 people on board the plane and only one survived and was captured.

But journalists who visited the scene Sunday said they saw two charred bodies in the cockpit of the wreckage of the propellor-driven aircraft, a third body in the fuselage, and a fourth body lying on the ground nearby.

They said they were told by military and other authorities that these were all the bodies found.

Sanchez Herrera said this was his second flight ferrying supplies for the Contras and that he was paid the equivalent of $48 in Honduran currency for each mission.

Two Americans, Eugene Hasenfus of Marinette, Wis., and James Jordan Denby of Carlinville, Ill., have been shot down over Nicaragua.

Hasenfus, whose plane was downed Oct. 5, 1986, was convicted in 1986 of flying supplies to the Contras but was pardoned by Ortega after serving less than 90 days of a 30-year sentence.

Denby, shot down Dec. 6, is awaiting trial in Managua on charges of crimes against public security and acts to submit the nation to foreign domination.

Denby, who owns a farm in Costa Rica, says he occasionally provided non- lethal aid to the Contras, but denies ferrying supplies to them.