PHNUM THMEI, Cambodia (AP) _ Communist Khmer Rouge guerrillas have retaliated against Vietnam's attempt to pin them against the Thai border by driving deeper into Cambodia and attacking Hanoi's troops from the rear, Thai military officers and guerrilla leaders said Saturday.

Unlike fighters of the Khmer People's National Liberation Front, who are seeking sanctuary from Hanoi's dry season offensive along the Thai-Cambodian border, the Khmer Rouge have sliced past Vietnamese vanguards, staged ambushes and are trying to stall the Vietnamese until monsoon rains bog down their heavy weapons, said a senior Thai military officer who asked not to be identified.

Soldiers from communist Vietnam invaded Cambodia in late 1978 and ousted the pro-Peking Khmer Rouge regime, which killed hundreds of thousands of Cambodians during its four-year rule. Hanoi launched an offensive against Cambodian rebels last December.

Khmer Rouge leader Khieu Samphan, speaking at the Phnum Thmei enclave about 12 miles southeast of the key Thai border town of Aranyaprathet, said, ''The Vietnamese are trying to prevent us from going inside, but they have failed. We still remain and are moving forward.''

Samphan claimed the Khmer Rouge in recent days had raided the provincial capitals of Siem Reap and Battambang in northwestern Cambodia. He said the guerrillas, undaunted by the Vietnamese offensive, had staged their deepest penetrations of the Cambodian interior since the guerrilla war broke out six years ago.

An Associated Press reporter, who traveled to Khao Noi about nine miles south of Aranyaprathet, heard sporadic mortar shelling over a three-hour period Saturday as the Khmer Rouge struck Vietnamese positions in the Phnom Yai Som area a few miles away. Thai officers said Khmer Rouge guerrillas repulsed a Vietnamese attack on Khmer strongholds in Phnom Malai on Friday, killing or wounding an estimated 100 Vietnamese. No independent confirmation of the reported attacks on Siem Reap and Battambang was available.

Samphan was in Phnum Thmei to welcome Prince Norodom Sihanouk, who crossed the Thai-Cambodian frontier into the war zone to the cheers of ''Hail the Prince 3/8'' and applause from about 1,000 Cambodian villagers.

Sihanouk, who heads a three-party resistance coalition which includes the Khmer Rouge and the Khmer Front, vowed to fight until Cambodia was liberated from the Vietnamese.

Sihanouk declared, ''We are united together, the patriots, the people. We are going to fight against the Vietnamese until the day we 100 percent liberate our nation.''

Sihanouk, a former Cambodian head of state, said the three coalition partners now are fielding more than 80,000 fighters against a Vietnamese force of 200,000 in Cambodia.

The purpose of his two-hour visit was to receive ambassadors' credentials from Senegal, Bangladesh, Mauritania and North Korea to his coalition government of Democratic Kampuchea, which is recognized by the United Nations.

Phnum Thmei is one of the best protected areas in the Phnom Malai region, where Vietnamese forces over the past two weeks have sought to destroy a string of Khmer Rouge bases, weapons caches and civilian concentrations.

The senior Thai officer said Vietnamese ground forces have not been able to penetrate Khmer Rouge enclaves.

''It has been impossible for the Vietnamese to move directly into the core area. So they shell long-range on both wings,'' the officer said.

The Thai officer said the Khmer Rouge hoped to maintain this ''stalemate'' until rainy weather in two months mires Vietnamese supply lines, forcing Hanoi's forces to retreat.

The Vietnamese, the officer said, are trying to trap the guerrillas against the Thai border by advancing against Phnom Malai from the east and against another stronghold, Khao Din, from the south.