WASHINGTON (AP) _ The United States has taken its complaint about Japan's beef and citrus import restrictions to GATT, where the Reagan administration's chief trade negotiator says the dispute may be settled by the end of this year.

Moreover, U.S. Trade Representative Clayton Yeutter says Zenchu, Japan's Central Union of Agricultural Cooperatives, had better rethink a threat made to boycott imports of American feed grains or suffer the consequences.

Yeutter predicted Wednesday at a news conference that the Japanese restrictions on beef and citrus imports will be declared illegal by GATT, the General Agreement on Tariffs and Trade. Such a finding would open the way to compensation or possible U.S. trade reprisals or a combination of both, he said.

Japan accepted the process at a GATT meeting in Geneva hours after breakdown of talks in Washington on Tuesday night. A panel to consider the issue is expected to be convened soon.

However, Yeutter said overall trade relations between the two countries are ''still very good'' and the United States ''would be happy at any time'' to listen to new Japanese proposals provided they ''reflect a whole lot more market liberalization than surfaced in the last few days.''

The Commerce Department has estimated that annual U.S. beef exports to Japan of about $550 million could grow to about $1 billion under free trade and exports of oranges and orange juice, now less than $100 million a year, could top $150 million.

''We will not permit this case to spill into the next administration,'' he said. ''We're going to resolve this dispute one way or another during this administration.''

Disclosing new details of his negotiations with Japanese Agriculture Minister Takashi Sato, however, Yeutter said the United States would have accepted a proposal for a three-year phase out of the beef quota if Japan had not sought to raise new barriers.

He also said a two-year phase out of orange and orange juice import quotas could have been accepted, but Japan insisted on five years and offered only a 10 percent annual tonnage increase.

Yeutter was asked about a threat by Zenchu to boycott U.S. feed grains because of the American request for GATT action on Japan's import restrictions.

''I would certainly hope (it doesn't happen), because that would be met with a harsh reaction by everyone here in the United States,'' including the Agriculture Department, the Office of the U.S. Trade Representative, the Reagan administration as a whole, and Congress, he said. ''Japan, as a nation, would pay a very high price for that kind of vindictive action.''

At a later telephone hookup with farm broadcasters, Yeutter reiterated his view that a GATT decision on the U.S. complaint would come relatively soon and said he made that clear to the Japanese agriculture minister.

''I told him that in no way would we permit this issue to spill over into the next administration, which is obviously what the Japanese would prefer,'' Yeutter said. ''So they're going to have to face it, at the very latest, toward the end of this year.''

Yeutter added: ''We're rather confident (of) winning the GATT dispute on both beef and citrus, and we're confident that the panel can come down with a recommendation and a decision in a relatively short period of time.''

The National Cattlemen's Association supported the move to put the issue in GATT because Japan's position left ''no choice'' to do otherwise, said association president Dale Humphrey.

''We cannot accept a proposal that would leave Japan's beef market essentially closed to producers,'' he said. ''Japan has a responsibility as a major trading partner to open its market. This is an opportunity for Japan to demonstrate that willingness and settle the beef trade dispute once and for all.''

John Datt of the American Farm Bureau Federation expressed disappointment that Japan ''has again refused to eliminate clearly illegal import restrictions.''

Datt said Japan's offer to phase out its import quota system in favor of new permanent import levies would be ''just replacing one protectionist program with a worse one. For a country's that's a major player in world trade, Japan is refusing to act like one.''