TOKYO (AP) _ The Reagan administration, embarrassed in its summit negotiations by the continued presence of U.S. oil firms in Libya, today fixed a June 30 deadline for the companies to pull out.

''They will be out one way or the other,'' Secretary of State George Shultz declared, adding: ''They may just have to abandon their assets.''

And Treasury Secretary James A. Baker III reinforced the warning. ''I think it is appropriate at some point to say to U.S. companies that are still there: you have had sufficient time. We've tried to be as lenient as possible.''

At the same time, though, the number of Americans working in Libya has increased sharply. A U.S. official, who demanded anonymity, said there were 500 to 800 in the country, some of them working for the oil firms.

President Reagan ordered all Americans out in January. He threatened they would be punished if they remained.

A few months ago, only 100 Ameicans were reported by the State Department to still be in Libya. But the U.S. official said the number had gradually built up since then.

Baker conceded it was hard to persuade Western European allies to quit buying oil from Moammar Khadafy as long as U.S. companies are still in Libya.

''I think the United States has to be able to make the point to its allies, if we're asking them to take action, that there are no longer United States' companies operating in Libya with the consent of the U.S. government,'' Baker said.

He said that, following the April bombing raid on Libya, some of the companies ''contacted us and suggested themselves it was time to walk.''

Neither Shultz nor Baker mentioned the June 30 deadline specifically. But other administration officials said that licenses permitting the firms to operate temporarily in Libya would not be renewed when they expire on June 30.

Robert Oakley, head of the State Department's anti-terrorist division, said the remaining U.S. firms should consider leaving even before that date.

''They've had their transition period,'' Oakley said.

Five U.S. oil firms have remained in operation in Libya despite Reagan's edict last January ordering all Americans out of Libya. They are Conoco, W.R. Grace, Amerada Hess, Marathon and Occidental.

The Reagan administration gave the oil firms, and another six oil-related service firms, special licenses to remain in business to give them time to phase out operations and sell off their assets.

Departure of the remaining U.S. firms ''will be a matter of weeks rather than days,'' Oakley told reporters.

At the time the special permits were granted, the admistration argued that requiring the firms to walk away from their holdings abruptly would in effect deliver a windfall of as much as $1 billion in assets to Khadafy.

Baker, in an interview on NBC's ''Today'' show, said, ''I must remind you that when we imposed these sanctions, we seized a substantial amount of Libyan assets, and if he (Khadafy) expropriates our oil companies interests over there we'd hold his assets as security.

Italian Premier Bettino Craxi, whose nation imports more oil from Libya than any other member of the 12-nation European Economic Community, in particular mentioned the continued presence in Libya of U.S. firms.

The statement on terrorism issued on Monday by the summit leaders made no mention of a boycott of Libyan oil, despite earlier U.S. hopes that one might be included.

''Between January and now the political situation has changed with Libya. We felt the time had come. The transition period is over,'' Oakley said.