BELGRADE, Yugoslavia (AP) _ Ayatollah Ruhollah Khomeini is taking three Yugoslavs to court for criticizing his order that author Salman Rushdie be killed without trial as a blasphemer, the daily Borba reported Friday.

Police in London, meanwhile, warned bookstores and publishers Friday night to be on guard against possible new arson attacks by Moslem militants over Rushdie's novel ''The Satanic Verses.''

Borba said the lawsuit was filed with the Belgrade district attorney's office against Stanislav Marinkovic, editor of Borba; Vesna Roganovic, a journalist for the paper, and Slobodan Selenic, a popular Yugoslav author.

Selenic, head of the Yugoslav Writers' Union, was sued because of a statement to Borba last month about ''Khomeini's frightening threat'' against Rushdie, Borba said. Marinkovic and Roganovic were cited for allowing the novel to be published, the newspaper said.

The charges are based on Article 158 of the Yugoslav criminal code, which forbids ''public humiliation'' of foreign countries or their leaders. Violation of the code carries a jail sentence of up to three years, Borba said.

A Yugoslav secretary who answered the telephone at the Iranian Embassy said no one was available for comment. Friday is the Moslem Sabbath and the holy month of Ramadan began a week ago.

When or whether Khomeini will have his own day in a Yugoslav court was not clear. Borba said several Yugoslav lawyers were representing Khomeini.

Among parts of ''The Satanic Verses'' offensive to Moslems a bordello scene in which prostitutes take the names of Mohammed's wives, and the suggestion that the prophet wrote the holy Koran instead of receiving it from Allah.

Two London bookstores in central London were set afire last Sunday in an apparent campaign of intimidation aimed at stopping sales of the book.

Sales have rocketed as a result of worldwide publicity over the novel.

London detectives refused to say whether more threats of violence have been received, but Scotland Yard said in a statement Friday night: ''Last Sunday's incidents may be the start of a campaign against premises which sell, publicize or make available Salman Rushdie's novel 'The Satanic Verses.'''

Rushdie, a 41-year-old British citizen born in Bombay, India, to Moslem parents, went into hiding after Khomeini issued his execution order Feb. 14.

Moslem critics of the Iranian patriarch's action say Rushdie should be tried in an Islamic court and given a chance to repent.