SALT LAKE CITY (AP) _ The Environmental Protection Agency has denied American Methyl Corp. permission to sell its methanol fuel additive for use in gasoline.

The company, formerly Anafuel International, has based almost all of its business since 1981 on the sale of ''Petrocoal,'' a methyl alcohol fuel additive produced from coal that boosts the octane rating of gasoline.

Ronald Eames, American Methyl's president, said the ruling ends the company's five-year fight with the federal government.

''Being a small company, we just do not have the financial resources to pursue the matter any further,'' he said. ''American Methyl is now out of the methanol business and we'll be trying to concentrate our efforts in other areas.''

In its denial, the EPA said the company failed to show that its additive would not cause cars to pollute excessively.

Petrocoal also failed to satisfy EPA concerns about whether cars would be harder to drive and would suffer deterioration of fuel system components, according to Barry Nussbaum, chief of the EPA's fuels branch.

The EPA originally granted American Methyl a waiver in 1981 under the Clean Air Act which allowed the fuel to be sold. That act recognizes that automobiles are designed to run on gasoline. Other types of fuels, such as methanol, must receive an EPA waiver before they can be sold for use in automobiles, Mr. Nussbaum said.

The EPA later tried to revoke the waiver but American Methyl challenged the action and won a decision from the U.S. Circuit Court of Appeals for the District of Columbia. The court ruled the EPA could not arbitrarily revoke a waiver once it was granted.

But a second lawsuit brought by the Motor Vehicles Manufacturers Association challenged the EPA's granting of the American Methyl waiver by claiming it was granted based on faulty test data. That challenge resulted in the appeals court siding with the association last year, according to Mark Slywynsky, senior attorney for the association.

The appeals court's decision on that suit led directly to the EPA revoking the waiver last week, Nussbaum said.

Eames said American Methyl sold close to $1 million of petrocoal in 1985, or enough additive for use in 200 million gallons of gasoline. Last April the EPA ordered sales suspended until it made its final decision on the waiver, he said.