NEW ORLEANS (AP) _ Baylor College of Medicine illegally kept Jews from participating in a program at a Saudi Arabian medical complex used by the Arab nation's royal family, a federal appeals court ruled today.

The 5th U.S. Circuit Court of Appeals upheld a lower court ruling that the Houston school discriminated illegally against Jews on its staff, but said the $280,000 ordered for attorney's fees was far too high.

The suit involved a program under which Baylor cardiovascular surgeons, anesthesiologists and other operating room personnel are sent for three-month stints to King Faisal Hospital in Riyadh, Saudi Arabia.

The hospital's main job is to care for and treat the Saudi royal family, but its services also are made available to Saudi citizens afflicted with particularly difficult illnesses, the court said.

The program is a popular one, it said, both because doctors get a chance to treat conditions found less often in the United States and because their salaries in Riyadh are nearly twice the salaries they earn in Houston.

The suit was filed in 1982 by Drs. Lawrence Abrams and Stewart Linde, anesthesiologists who wanted to work at Faisal Hospital but were not allowed to.

Both were told that, because they were Jewish, they would not be able to get entry visas for Saudi Arabia, the court said.

''There is no evidence in the record that the statement represented the actual position of the Saudi government with regard to the participation of Jews in the program,'' the 5th Circuit said in a unanimous ruling.

''In addition, there is no evidence that Baylor even attempted to ascertain the official position of the Saudi government on this issue,'' it added.

The 5th Circuit noted that Dr. Michael DeBakey, head of the medical school, testified that he occasionally had sent Jewish physicians to see special patients in Saudi Arabia, and had never had any trouble getting visas for them.

The judges split 2-1 on the issue of attorney's fees. The majority said U.S. District Judge James DeAnda failed to consider whether lawyers had duplicated effort, and whether all of their time was spent in legal work, or whether some was clerical work.