NEW YORK (AP) _ An ex-Army officer who said he lost his Vietnam War command for reporting atrocities by American troops was not libeled when CBS broadcast a story challenging his account, a federal appeals court ruled Wednesday.

The 2nd U.S. Circuit Court of Appeals threw out the libel suit first filed in 1974 by retired Lt. Col. Anthony Herbert, a decorated war veteran whose sensational charges brought him brief fame in the early 1970s.

Herbert had no grounds for his suit against CBS, its ''60 Minutes'' correspondent Mike Wallace and producer Barry Lando, the three-judge appellate panel unanimously agreed. Neither the broadcast, which appeared in February 1973, nor an Atlantic Monthly magazine article by Lando that was published three months later were actionable, the court said.

Herbert served in Vietnam from 1968 to July 1969 as commander of the 173rd Airborne Brigade. He was relieved of his command and sent back to the United States.

He waged a lengthy battle to recover his command, charging that superiors ordered his removal because he reported instances of American troops brutalizing unarmed Vietnamese. The Army rejected his claims and Herbert retired in 1972. He then wrote a book, ''Soldier,'' in which he brought his case to the public eye.

The ''60 Minutes'' segment was titled ''The Selling of Colonel Herbert.'' Wallace, who narrated the segment, said CBS was unable to verify most of Herbert's account, pointed up many purported discrepancies in his story and said it appeared that Herbert only raised the war crimes allegations after being relieved of his command.

A CBS spokesman, James Noonan, said network officials were delighted by the ruling but had not had time to examine it. Wallace and Lando were unavailable for comment, he said.

Herbert's lawyer, Jonathan W. Lubell, said the decision was disappointing and may be appealed. He said he had been unable to reach Herbert, who lives in Colorado.

As a public figure pressing a libel case, Herbert had to demonstrate that CBS, Atlantic Monthly, Wallace or Lando published false, defamatory statements about him with ''actual malice'' - a legal term meaning that they either knew the statements were untrue when published or that they acted recklessly.

His suit zeroed in on 11 statements in the broadcast and magazine article that, his lawyers said, met those criteria.

But in October 1984 U.S. District Judge Charles S. Haight Jr. ruled that nine of the 11 statements were either demonstrably true or that there was no evidence any of malice in publishing them. He dismissed all of Herbert's case against Atlantic Monthly and narrowed the remaining case to just two statements, one in the television broadcast and one in Lando's article.

Haight said a jury would have to consider those two statements to decide if they are libelous. But the appellate court disagreed, saying that the statements applied only to ''subsidiary'' issues and could not, of themselves, be libelous.

Though Wallace may have erred in saying that ''to a man'' Herbert's Vietnam War acquaintances had denied that he ever mentioned atrocities while in Southeast Asia, and Lando made another minor factual mistake in the magazine article about Herbert three months later, the mistakes did not change the overall tone of the stories attacking Herbert's claim to have reported war crimes, the appeals court said.

Herbert could ask the full 11-judge 2nd Circuit or the U.S. Supreme Court to reinstate his case.

The protracted litigation has reached the Supreme Court once before. In 1979, the high court overruled the 2nd Circuit and ordered CBS to turn over film clips that were not broadcast to Herbert's lawyers to enable them to determine what information was available to the network at the time of the broadcast. The 2nd Circuit had agreed with CBS that the forced turnover of such material would be an unconstitutional intrusion into the editorial process.