MOSCOW (AP) _ The official Tass news agency said Friday the United States is pursuing gunboat diplomacy against Libya by ordering naval exercises off the Libyan coast and is balancing ''on the brink of a military conflict.''

It was the first published Soviet comment since the U.S. Navy announced on Thursday that a battle fleet, including the aircraft carriers Saratoga and Coral Sea, would conduct flight operations in ''international air space'' in the Mediterranean through Jan. 31.

Libyan leader Col. Moammar Khadafy denounced the U.S. maneuvers as ''an aggressive provocation'' and said he had placed his navy and air force on full alert.

The United States has accused Khadafy of backing international terrorism and of suporting the Abu Nidal terrorist group blamed for the Dec. 27 attacks at the Rome and Vienna airports in which 20 people, including five Americans, were fatally wounded. The Reagan administration ordered economic sanctions against Libya earlier this month.

In his article, Tass commentator Vasily Kharkov wrote, ''The large-scale U.S. naval maneuvers which have begun in the Mediterranean and involve more than 25 ships ... are an obvious provocation against sovereign Libya.''

''The U.S. muscle-flexing off Libyan shores is a dangerous relapse into gunboat diplomacy which has long become a symbol of lawlessness and violence,'' he said.

''While groundlessly accusing sovereign Libya of aiding and abetting 'terrorism,' the present American administration itself demonstrates an example of terrorism institutionalized as state policy.''

Kharkov added, ''The U.S. administration continued to work up tension, employing blatant methods of power politics .... Finally, having clenched a big naval fist, Washington decided to recourse to extreme measures, balancing in fact on the brink of a military conflict.''

Although U.S. Defense Department sources said in Washington that no military strike is planned against Libya, there is a long-standing dispute between the countries over the Gulf of Sidra, a U-shaped body of water more than 100 miles in length. Libya claims the gulf, and the United States refuses to recognize that, asserting Libyan territorial waters extend only 12 miles offshore.

Two Libyan warplanes were shot down by U.S. Navy jets in a clash over the gulf in August 1981.

President Reagan imposed economic sanctions against Libya on Jan. 7 and then froze Libya's assets in the United States and ordered the estimated 1,500 Americans in that North African country to leave by Feb. 1.