WASHINGTON (AP) _ A treaty to sharply reduce strategic nuclear arms is possible by the end of the Reagan administation, especially if the Soviets drop efforts to curtail the U.S. ''Star Wars'' missile defense program, a top negotiator said Friday.

''The prospects are rather hopeful because the Soviets have things they can give up,'' said Edward L. Rowny, referring to heavy Intercontinental Ballistic Missiles that the administration says are destabilizing. ''We would be able to improve security on both sides.''

U.S. officials are studying proposals presented in Washington last week by Soviet Foreign Minister Eduard A. Shevardnadze that would attempt to limit Star Wars research by defining the technical parameters of the weapons under research, Rowny said.

Rowny, special adviser to the president for arms control, spoke at a seminar sponsored by the conservative Ethics and Public Policy Center in Washington.

A proposed deal to reduce superpower nuclear arsenals by 50 percent foundered at the Iceland summit last October over Soviet insistence that the U.S. administration limit Star Wars testing to the laboratory for 10 years.

President Reagan described the Soviet demand as an effort to tighten the 1972 Anti-Ballistic Missile Treaty, which limits missile defenses. Soviet officials later softened their stand, saying privately that they might accept some tests in open-air laboratories or even in space if both sides agreed to abide by the ABM pact for a decade.

Shevardnadze ''recast some of the linkage'' between restricting Star Wars and cutting nuclear weapons arsenals, ''but it is still there,'' said Rowny.

During his visit, Shavardnadze repeated his insistence that both sides abide by the treaty for a decade, and proposed ''parameters'' for testing under Star Wars, formally known as the Strategic Defense Initiative, said Rowny.

Publicly ''they don't want to describe in detail what their definition is (of the ABM pact), and we are studying those parameters,'' said Rowny. ''If they provide a way that we can have research without crippling SDI, then we might accept it.''

''Now, they are in effect saying that you don't have to restrict research to the ground, but must keep it within these parameters,'' Rowny said. ''It's getting at the problem through technical means.''

Rowny declined to discuss specific details. But other officials say the Soviets are seeking to put limits on the brightness of lasers under research or the speed of collision weapons being tested.

Pentagon officials have told congressional committees that it might be possible to orbit weapons that destroy enemy missiles by colliding with them as early as 1994. More exotic devices, such as lasers and particle beams, could take decades to reach the weapons stage, according to SDI scientists.

Asked why the administration objected to remaining within the ABM pact for 10 years when it may take that long to develop effective SDI defenses, Rowny said: ''If (the Soviets) said we should abide by ABM for 10 years, that would be one thing.''

''They are saying that 'You can't do any research that would lead to a deployment in 10 years.' They haven't said, 'You can do all the research you want and deploy at the end of 10 years.'''

Rowny also:

-Said the treaty to eliminate superpower Intermediate Nuclear Forces could make Western Europe less secure if NATO does ''not redress the conventional weapons imbalance.'' Pentagon officials say the Kremlin-led Warsaw Pact has a substantial edge over the Atlantic Alliance in non-nuclear weaponry.

-Repeated administration assertions that a Soviet radar facility at Krasnoyarsk, Siberia, violated the ABM treaty, although Kremlin officials allowed U.S. congressmen to visit the facility. The administration argues that the radar violates the pact because it is too far from the Soviet border and points inward rather than outward.

-Defended the verification measures of the proposed INF pact. The administration relaxed some of its demands after the Soviets agreed to eliminate all their medium-range nuclear missiles, rather than keeping 33 of them in Asia. And Rowny said that the proposed treaty allows adequately strict on-site monitoring of the dismantling of the weapons as well as inspection of ''suspect sites'' on demand.