WASHINGTON (AP) _ The 39 parties seeking control of 13 broadcast stations operated by RKO General Corp agreed Thursday to use a federal mediator to try to end 21 years of litigation over the licenses.

The 38 groups, including groups owned in part by entertainers Bill Cosby and Arlene Francis, have tied up the 39th, RKO, in a protracted tug-of-war that has kept the company from selling or buying any of the stations whose licenses together are worth an estimated $280 million.

Rather than face what FCC lawyer Alan J. Schneider called ''many more years'' before the FCC and the courts, the parties agreed to accept a Federal Communications Commission offer to allow a mediator to negotiate a settlement.

None of the parties was under any obligation to join the talks, but all agreed to do it.

At a meeting at FCC offices, they elected James C. McKinney, the FCC's mass media bureau chief, as mediator.

Following the two-hour meeting McKinney expressed confidence ''We are going to settle all 13 properties by Jan. 31,'' the deadline established by the commission.

Actually, he said, a deal by Dec. 31 is more likely because of more favorable treatment of such deals under tax laws likely to change dramatically in 1987.

RKO, a subsidiary of Akron, Ohio-based GenCorp, sent its chief executive officer, A.W. Reynolds, to tell the group it is willing to settle to end the battle to keep the stations.

McKinney said Reynolds expressed the company's total commitment to work in the process ''that would hopefully see all of the RKO properties move to someone else.''

An RKO spokesman later quoted Reynolds as telling the group, ''If we do have to remove ourselves from the business it is with great reluctance.'' Spokemsan Steve Ellis said GenCorp. was seeking protect its shareholders by trying to obtain fair market value for the stations.

GenCorp could lose all the licenses if the commission finds it unfit to be a licensee because of previously found RKO misrepresentation of a Securities and Exchange Commission investigation. That was an investigation of GenCorp. when it was known as General Tire & Rubber Co.

There was ''a general feeling of optimism in the room,'' McKinney said. In addition to lawyers, he said several company principals came to the meeting. ''We made a great deal of progress,'' he said.

Some of the groups immediately began negotiations and all of the groups promised to report progress to McKinney by Oct. 3.

McKinney said six companies not already a part of the lengthy proceedings had offered to negotiate to buy all or some of the stations. But the 39 said they would prefer to deal among themselves before bringing the others into the talks, he said.

He refused to name the lawyers for the six companies who contacted him and said he did not know who they represented.

In any of the 13 cases any one party to the negotiations could refuse to deal and scuttle the entire settlement.

Cosby, as partner in Cozzin Communications, has applied for all of the licenses, as has South Jersey Radio.

Miss Francis is a partner in Women Owning Radio, which has applied for stations in New York City.

The RKO stations involved in the negotiations are WHBQ-AM and TV, Memphis, Tenn.; WOR-AM and WXLO-FM, New York; KHJ-AM and KRTH-FM, Los Angeles; WGMS-AM, Bethesda, Md.; WGMS-FM, Washington; WRKO-AM and WROR-FM, Boston; WAXY-FM, Fort Lauderdale, Fla. KFRC-AM, San Francsico and WFYR, Chicago.

Broadcasting Investment Analysts, a Chantilly, Va., broadcast appraiser used industry statistics to place the value of the stations at $279.7 million. The estimate, prepared for Broadcasting magazine, was made without access to RKO financial records.

Two other RKO stations, KHJ-TV, Los Angeles and WOR-TV, Seacaucus, N.J. are not involved in the new negotiations.