NEW YORK (AP) _ A former editor for The Wall Street Journal sued the financial daily Tuesday for $12.64 million, claiming he was fired and his reputation smeared by a false charge of plagiarism.

Jonathan Kandell, the Journal's assistant foreign editor prior to his dismissal May 2, also demanded the return of original notes and other documents that he contended the paper's editors had kept, knowing they would support his position.

The Journal fired Kandell, 43, after accusing him of lifting uncredited material from another writer's book for a page 1 article about three private entrepreneurs in eastern Europe.

Kandell quoted Paul Steiger, deputy managing editor of the Journal, as having said his alleged offense was ''the most serious transgression to take place at the Journal since the Foster Winans case.''

R. Foster Winans was convicted in 1985 of using his position as a Wall Street Journal columnist to profit from insider stock trading information. He was sentenced to 18 months in prison, a $5,000 fine and five years' probation.

In his lawsuit, filed Tuesday in U.S. District court in Manhattan, Kandell contends he was libeled and his professional reputation destroyed by the Journal. He demanded $10 million in punitive damages and $2.64 million in compensation for ''willful, wanton and reckless disregard of the truth'' and ''actual malice'' on the paper's part.

He said the libel was contained in a letter of apology that the newspaper sent to the author of a book dealing with the same subject and in a subsequent article in the weekly newspaper Village Voice about Kandell's case. The Voice was not named as a defendant in the suit.

Roger May, a spokesman for Dow Jones & Co. Inc., owners of the Wall Street Journal, said, ''Dow Jones believes the suit is utterly without merit and the company intends to defend itself vigorously.''

Kandell's article, in the March 30 edition of the Journal, profiled a Soviet machine tool maker, a Czechoslovak inventor and a Hungarian banker as examples of individuals succeeding in the private sector.

Kandell said he based the story on interviews with the three individuals in their countries after hearing them speak at a meeting in New York last November, sponsored by the Council on Foreign Relations.

He said that at the same meeting, he also learned of a book, ''Communist Entrepreneurs,'' and met the author, John W. Kiser III. He said he later ''read the book once and put it aside.''

Kandell's suit said he told Steiger and senior editor Barney Calame that he hadn't relied on Kiser's book and hadn't credited it because he had ''done all his own legwork and the book was not where he had first learned of the subject.''

On May 1, Kandell said, the editors told him they had decided he should have mentioned Kiser's book in the article, ''but emphasized that it was a judgment call.''

Later that same day, Kandell said, he learned he was to be fired. He quoted Steiger as saying he had shown a ''lack of candor'' and his replies to questions had left him (Steiger) ''with a gut feeling that I had not been truthful.''

He said he was dismissed the next day, effective immediately, after consulting with his lawyer and refusing to sign a letter of resignation that he said would have kept him on the Journal's payroll through June 30.