NEWPORT, R.I. (AP) — The Vanderbilt family, once synonymous with American wealth and power, has fallen into a full-blown public spat with the organization that now owns their spectacular Rhode Island mansion.

The conflict includes intimations that the group might sue, or that it might evict the two Vanderbilts who still summer on the third floor of the house, called The Breakers; even family member Anderson Cooper has not been spared from the fray.

"I'm waiting for them to throw my clothes out the window," said Gladys Szapary, the great-granddaughter of the man who built the 70-room mansion in 1893, who has summered on the third floor her entire life.

In the late 1940s, her grandmother, Countess Szechenyi, agreed to lease the downstairs for $1 per year to the Preservation Society of Newport County, then a fledgling group that was trying to save the city's famous but vacant Gilded Age mansions from the wrecking ball.

Even in a city filled with over-the-top mansions, The Breakers was like no other. Cornelius Vanderbilt II, president and chairman of the New York Central Railroad, spared no expense when he built it. It has wall panels made of platinum.

The countess and her family moved into rooms on the third floor that had been built for her brothers. She turned a housekeeper's room into a kitchen. They put up a baby gate on stairs between the second and third floors to keep tourists out.

In 1972, Szechenyi's heirs sold the house to the Preservation Society for $366,000. But her daughter, Countess Szapary, was invited to stay. After Szapary's death, her children, Gladys and Paul Szapary, were invited to remain on the third floor, rent free.

"It will be helpful to us to be able to tell our visitors that the original owners' great-grandchildren continue to live in the house," the Preservation Society's board president wrote in a letter to the pair in 1998.

Paul, 65, and Gladys, 62, said they feel a responsibility. And they, and others in their family, believe the home is not being run properly.

Their objections coalesced around a proposal put forth by the Preservation Society to build a visitor center on the grounds. The society said it would be tucked in a little-used portion of the 13-acre estate and would provide a sheltered and handicapped-accessible place to buy tickets, use the bathroom, and purchase snacks and sandwiches.

But many neighbors and preservationists objected, saying it would detract from The Breakers' status as a National Historic Landmark. They argue the proposal is an example of the society sacrificing its mission to preserve history as it hunts for new ways to make money. The group is Rhode Island's largest cultural institution by revenue, bringing in $20 million in the most recent fiscal year.

Last month, 21 members of the Vanderbilt family, including CNN journalist Cooper's mother, designer Gloria Vanderbilt, decided to take public action. They wrote to the group's board to express their concern about the proposal, which they say is symptomatic of the group's dictatorial management style. They said they won't donate money or family objects to the nonprofit under "the current leadership climate," which they said has alienated many longtime supporters and top donors, Vanderbilt or no.

The president of the Preservation Society's board fired back in a memo that the signers had contributed only $4,000 to the group in recent years, and that most of the family members' items displayed at The Breakers were not very important, or even "minor," such as hairbrushes or wastebaskets. He made a veiled threat against the Szaparys, saying their occupancy "can be ended at any time."

The Szaparys call his response demeaning and misleading, saying it brushes off what has been decades of generosity from the Vanderbilts.

Soon after, the Preservation Society's lawyer threatened to sue a group the Szaparys belong to called Preservation Society Friends, which opposes the visitor center and is otherwise critical of the society's management. The lawyer accused the group of "collaboration with the Vanderbilt family" to try to stop the visitor center "at any cost."

The Preservation Society's leadership declined an interview request. They sent a written statement that opposition to the visitor center was by a small and vocal minority and that it was unfortunate the debate has become "personal and unpleasant." They didn't answer questions about possible litigation or eviction.

The Szaparys and Preservation Society Friends say if there is unpleasantness, it is emanating from the Preservation Society.

Gladys said, for example, that she was asked by Helen Winslow, widow of the society's former board chairman, to arrange for Cooper, her famous cousin, to deliver the group's annual Winslow Lecture. Cooper had visited The Breakers as a child and agreed to speak but was snubbed with no explanation, she said. Instead, the lecture focused on "Great Houses of Havana."

Cooper would not comment. The Preservation Society said it would be delighted to have Cooper speak but did not explain why it ignored his offer.

At the Preservation Society's annual meeting this month, the name Vanderbilt was not uttered.

The Szaparys live in New York when they're not staying at The Breakers, and both say their fight is not about retaining their residency on the third floor. If they are asked to leave, so be it, they said. What they care about most is keeping history alive, and preserving their family's legacy.

"The Breakers was a house. It's not just a mansion, a museum. Children played there. People got sick there. All the things that families undergo happened there. We think that people who visit there are interested in that aspect," Paul said.

His sister chimed in: "There was a family that lived there."