ATLANTIC CITY, N.J. (AP) _ The Fort Worth Star-Telegram took top honors in the public service category of the National Headliner Awards and a Detroit Free Press reporter won the investigative reporting category.

The Star-Telegram's articles, reported by Mark Thompson, led to a U.S. Army probe into a design flaw in military helicopters that caused the death of 250 servicemen over 17 years, contest officials said Wednesday.

Thompson detailed a phenomenon known as ''mast bumping,'' in which the rotor blades of the helicopter can chop through the passenger compartment.

Dolly Katz of the Detroit Free Press won for a series of articles on the cases of 205 physicians who went before a state regulatory panel between 1977 and 1982 for alleged violations of Michigan laws.

The series led to a change in Michigan law regarding medical care, said Herb Brown, chairman of the 51-year-old National Headliner Awards committee.

The contest's judging panel, comprised of newspaper editors and publishers from across the country, also gave a special citation to the Los Angeles Times for outstanding endeavor in communications for its coverage of the 1984 Olympic games.

Brown said other winners chosen from among 1,200 total entries in the print division were:

- Outstanding news reporting by a newspaper with circulation below 50,000: Michele Lesie of The Journal, Lorain, Ohio, for ''Why Jenny Died,'' which told the story of a talented 17-year-old who committed suicide;

- Outstanding news reporting by a newspaper with circulation between 50,000 and 150,000: Greta Tilley, Greensboro News and Record, Greensboro, N.C., for a look inside the world of the mentally ill;

- Outstanding news reporting by a newspaper with a circulation over 150,000: William Marimow, The Philadelphia Inquirer, for a series which detailed stories of innocent civilians who were mauled by the city police department's K-9 patrol;

- Consistently outstanding local interest column: Joel McNally, The Milwaukee Journal, Milwaukee, Wis., for his column, ''The Innocent Bystander;''

- Consistently outstanding special column: Dennis McDougal, Los Angeles Times, for his column, ''On The Radio;''

- Consistently outstanding editorial cartoons: William Day, The Commercial Appeal, Memphis, Tenn.;

- Outstanding spot news photography: Peter Schlueter, Arizona Republic, Phoenix, Ariz.;

- Outstanding feature photography: Jay Koezler, The Rocky Mountain News, Denver, Colo.;

- Outstanding sports photography: Bob Langer, Chicago Tribune;

- Consistently outstanding sports column: Tom Archdeacon, The Miami News, for his column, ''Inside Moves;''

- Consistently outstanding feature writing: Susan Ager, Detroit Free Press;

- Outstanding coverage of a major news event by a magazine: Andrew Decker, ARTnews, New York, for ''A Legacy of Shame,'' which revealed that the Austrian government possesses art works looted by the Nazis;

- Consistently outstanding magazine column: William Taafe, Sports Illustrated, for his radio-television column;

- Outstanding syndicate spot news photography: Tom Harm, United Press International;

- Outstanding syndicate feature photography: Scott Stewart, The Associated Press;

- Outstanding syndicate sports photography: Dave Tenenbaum, The AP;

- Outstanding magazine color photography: Larry Downing, Newsweek;