LOS ANGELES (AP) _ When Nancy Reagan told old pal Merv Griffin of her post-White House dream to open a drug rehabilitation center bearing her name, ''every mover and shaker'' for miles around was called to action.

Rolls-Royces bearing such luminaries as Texas oilman Marvin Davis, media magnate Rupert Murdoch and actress Eva Gabor poured into Griffin's Beverly Hills estate last May for a power breakfast at his tennis pavilion.

But the self-described ''little people'' of Lake View Terrace, fearing the center proposed for the bankrupt Lake View Terrace Medical Center would invite crime, just said no - sparking a battle described as ''David versus Goliath.''

''I'm not fighting Nancy Reagan, I'm not fighting Ronald or what they stand for or what they do. I just don't think this is the place for it,'' said Suzan Kent, 28, who lives across the street from the medical center.

Mrs. Kent and others picketed a welcome-home gala for the Reagans earlier this month with signs reading, ''If you really care, put it in Bel-Air,'' referring to the Reagans' new neighborhood.

Lake View Terrace, about 15 miles north of Bel-Air, is a small town on the edge of a big city.

It sits at the foot of the Angeles National Forest, one of the last semi- rural havens for folks who prefer goats grazing next door to Ferraris cruising Sunset Boulevard.

When two planes collided over nearby Pacoima Junior High School in 1957, an accident depicted in the movie ''La Bamba,'' there was no place to take the dying. The residents rallied to build the medical center.

''So it holds a very special place in the hearts of the people,'' said Lewis Snow, an officer in the Lake View Terrace Home Owners Association.

Snow and others want another hospital, not a drug center. They fear the center's residents will target nearby homes when they need extra cash.

On the other side of the debate is Phoenix House, the nation's largest private, non-profit drug abuse agency. Founded in 1967, it operates 10 centers - six in New York and four in California - and is accustomed to opposition.

''If we had to back down every time a community said, 'Not here,' we'd never treat the drug problem in this country,'' said spokesman Christopher Policano.

When Mrs. Reagan decided to put her imprimatur on a center in the Los Angeles area, she asked Griffin to help, reportedly saying, ''I want to come back and go right to work.''

Griffin responded by inviting what he described as ''every mover and shaker in this part of the world'' to his May 2 breakfast.

About $4 million has been raised toward the estimated $10 million cost, including $1 million at the $25,000-a-table welcome home party thrown by Barron Hilton and Griffin.

The 15-acre site would offer treatment for 150 teen-agers who would live and attend school at the center, plus an additional 60 young adults over 18. There would be out-patient counseling for 15 to 30 youngsters who have just become involved with drugs, and Mrs. Reagan would have an office at the center.

The center would not accept anyone with a history of violence nor anyone on drugs when they were admitted. The agency boasts a 90 percent success rate among the 40,000 or so graduates of its program.

The Los Angeles Police Department has remained neutral, but area Capt. Tim McBride is willing to give Phoenix House a chance.

''We have a serious drug problem all across the country, and we've got to do something about that problem. The information I have about Phoenix House is that they're successful and they're well run,'' he said.

The center might open this year, but roadblocks remain.

Residents plan to pack a hearing before a city zoning officer on Friday. On Tuesday, the City Council will consider a request for an environmental impact report. That could mean an eight-month setback for the center.

Some residents believe the center will find a home elsewhere.

''We definitely have a David-versus-Goliath fight,'' said resident Jules Bagneris. ''And David did win.''