KARUIZAWA, Japan (AP) _ The defense chief said Friday that friction over trade could endanger the U.S.-Japan Security Treaty, under which the United States provides much of this country's military protection.

Security relations are excellent, but trade and economic ties have ''seriously deteriorated'' and ''repercussions could affect the U.S.-Japan defense relationship,'' said Koichi Kato, director general of the Defense Agency.

Under the security treaty, the United States must protect Japan against foreign aggression.

Washington has urged Japan in recent years to increase its defense effort. The United States spends about 7 percent of its gross national product on defense, but Japan has long maintained a ceiling of 1 percent of GNP for that purpose.

Kato made the comments in a panel discussion at an annual seminar of the ruling Liberal Democratic Party at this resort town 80 miles northwest of Tokyo.

The trade dispute stems partly from U.S. demands that Japan open its markets to overseas goods in order to cut the American trade deficit with this country, which reached $36.8 billion last year.

Prime Minister Yasuhiro Nakasone pledged in April that Japan will do so. The government is expected to announce next week the official ''action plan'' to remove bureaucratic trade barriers slowing the entry of U.S. exports into Japanese markets.

Kato said the link between trade and security is being taken seriously in the United States. Referring to statements by U.S. officials, he said, ''We must handle the trade friction issue carefully.''

The defense chief repeated earlier hints that the ceiling on defense spending would be removed.

Earlier Friday, the Cabinet formalized a decision to allow a 7 percent increase in defense spending in the fiscal year beginning in April 1986. Although the final amount must be approved by Parliament, such an increase probably would exceed the ceiling of 1 percent of GNP maintained since 1976.

Kato also noted that the 1 percent limit was established originally as a temporary measure when rising oil prices made it impossible to appropriate funds for five-year budgets, which had been the custom. Japan imports nearly all its oil.

The national attitude toward defense also must change, Kato added. The Japanese concept of defense is different from other countries because Japanese tend to associate the concept of ''military'' with ''anti-democracy,''h e said.

Since Japan's defeat in World War II, many Japanese have shown a distrust of the military. The nation's 1947 constitution renounces war as a sovereign right.