ATLANTA (AP) _ A highly commended state trooper cannot use a ''sixth sense'' to stop drug suspects on the highway, a federal appeals court has ruled.

Florida Highway Patrol Trooper Robert Vogel, 38, earned a reputation as one-man narcotics squad after for making 30 drug arrests between March 1984 and June 1985. The arrests netted nearly 3,000 pounds of marijuana and 36 pounds of cocaine.

One of his stops involved Timothy Andrew Smith, 27, of Bowie, Md., and Stephen Lawrence Swindell, 34, of Wheaton, Ill., who were charged with trafficking cocaine. Their rental car was stopped June 5, 1985, on Interstate 95 in Port Orange. A police dog sniffed out 2.2 pounds of cocaine in the car.

The men were not charged with a traffic violation. A federal judge in Orlando ruled that the cocaine seized in the arrests could be used as evidence, and both men were convicted.

But Tuesday, the 11th U.S. Circuit Court of Appeals overturned the convictions, saying Vogel lacked probable cause to stop and search the men's car.

''The stop was an unreasonable seizure under the Fourth Amendment,'' wrote Judge Phyllis Kravitch.

The decision follows a similar ruling by a Florida state judge. Last month, Circuit Judge C. McFerrin Smith of Volusia County ruled, ''Trooper Vogel obviously has a 'sixth sense' in discovering illegal drugs. However, he has been unable or unwilling to 'articulate' what makes up his 'sixth sense.' ''

Vogel has said he developed a list of certain characteristics to identify drug smugglers when he was working with federal drug agents last summer.

According to his profile, a drug smuggler usually is a black man in his early 30s who dresses casually and drives an out-of-state rental car north on I-95, at or below the speed limit, at about 3 a.m.

During a hearing on the Johnson case in March, Vogel presented charts and graphs to identify common characteristics in the 30 drug arrests he made last summer.

Vogel, who has been with the highway patrol 14 years, has been commended by the U.S. Drug Enforcement Administration and the Justice Department as well as the patrol.

On Monday he found 4 1/2 pounds of cocaine valued at $400,000 stashed in a trap door above the gasoline tank of a car driven by a New Jersey man. The man had been stopped for weaving on the highway.