BANGKOK, Thailand (AP) _ Opening of negotiations and relative quiet on the warfront have sparked a glimmer of hope that two decades of savage conflict in Cambodia can be solved through political compromise rather than more bloodshed.

After an 8-year-long stalemate, two rounds of peace talks were recently held between key protagonists: Cambodian resistance leader Prince Norodom Sihanouk and Prime Minister Hun Sen of the pro-Hanoi regime in Cambodia.

Vietnam, which invaded Cambodia in 1978, now says that if an acceptable political solution is reached, it may withdraw its troops before an announced 1990 pull-out. And Moscow has indicated it wants to heal itself of two festering Asian sores: Cambodia and Afghanistan.

Although negotiations in Paris last December and January did not produce a mutually acceptable blueprint for peace, both sides say they are working toward freedom from outside interference and an end to Cambodia's cycle of agony. Since 1970, Cambodia has suffered a devastating five-year war; a reign of terror by the Khmer Rouge; the Vietnamese invasion; and a continuing guerrilla war. Millions have perished.

Deep suspicions of Vietnam's motives remain, as do fears of a return to power of the brutal Khmer Rouge. And analysts are puzzled by the moves of the unpredictable prince, who on Jan. 30 resigned as head of a three-party resistance coalition and who regularly lashes out at his partners-in-war.

The 65-year-old Sihanouk - onetime king, consummate survivor, enduringly popular leader - headed a coalition of the communist Khmer Rouge, the Khmer People's National Liberation Front and his own followers fighting the Vietnamese. China arms the coalition. The Soviet Union fuels Vietnam's war machine.

Stressing that Vietnam was eager to attract investment from the West, Hanoi adviser Nguyen Xuan Oanh admitted recently the war in Cambodia was ''a very costly proposition. It has drained our resources.'' Besides the burden of keeping about 140,000 troops in Cambodia, the opportunity costs are great: most non-Soviet bloc nations continue to shun Vietnam, insisting on a troop withdrawal before engaging in full trade and aid relations with Hanoi.

Also widely seen as helping the process of political settlement are the rise of seemingly more flexible leaders: Mikhail Gorbachev in Moscow, Deng Xiaoping's followers in Peking, and Vietnamese Communist Party chief Nguyen van Linh in Hanoi. China cites the Vietnamese grip on Cambodia as a key barrier to detente with Moscow, which appears to be seeking closer ties with Peking.

Changes are also perceived in the military situation. ''I can't tell you who is winning the war in Cambodia ... but presently both sides are turning to political means instead of fighting on the battlefield,'' says Lt. Gen. Sihadaj Bunnag, spokesman of Thailand's military command.

Several Western analysts interviewed agree that in recent months fighting has decreased substantially, although reasons given for this are largely speculative.

Rather than seeking combat, one theory says, the resistance guerrillas are concentrating on grass-roots organization and propaganda, trying to solidify support for their respective groups in event action shifts to the political arena.

Militarily, the balance appears to be unchanged from recent years: the estimated 50,000-strong fighters of the coalition can't defeat the Vietnamese but would probably rout the weak, pro-Hanoi Cambodian army if Hanoi pulled out. The Khmer Rouge remains the most powerful military force within the coalition.

For many Western and Thai analysts a ''worst case scenario'' would be Vietnamese entrenchment in Cambodia; continued warfare; the indefinite existence of 250,000 Cambodian refugees along the Thai-Cambodian border.

Perhaps the most optimistic future envisioned by Western observers is a Vietnamese pull-out; a coalition government in Cambodia with Sihanouk as a major figure; and a withering away of the Khmer Rouge through defections of younger conscripts and general hatred of a force robbed of its major appeal: fighting a foreign invader.

Sihanouk himself outlines a ''quadripartite provisional government,'' including the three coalition members and the Hun Sen side. This foursome would work on organizing general elections under international supervision and be backed by a four-party national army.

Sihanouk appears to be key to the question of peace or war in Cambodia. He still enjoys widespread domestic popularity and by far the greatest backing internationally among the coalition's leaders. His latest maneuvers may be designed to get the fractious coalition solidly behind his negotiation efforts.

Since his resignation, the two coalition partners have come begging for his return. China, although the staunchest backer of the Khmer Rouge, stressed its support. Communist Party chief Zhao Ziyang praised Sihanouk as ''a great patriot and experienced politician.'' Should the coalition sides ever compete for power in Phnom Penh, Sihanouk would need Peking's help in curbing the Khmer Rouge.

---

EDITOR'S NOTE - Denis D. Gray, bureau chief for The Associated Press in Thailand, covered Cambodia during the Indochina War and visits the country from his base in Bangkok.