ATLANTA (AP) _ The U.S. Penitentiary in Atlanta where Cuban inmates set fires and took hostages was for decades one of the blacker holes in the federal prison system, a flashback to Jimmy Cagney movies full of bad eggs and tough cookies.

''Big A,'' as former residents knew it, still is no garden spot but a $63- million, five-year renovation project has been under way to make the 85- year- old facility a state-of-the-art prison.

A report by the U.S. Bureau of Prisons in 1978 suggested closing the prison because its outdated design and deteriorating condition made adequate security impossible.

It was ordered closed by the fall of 1984, but the government started using it to house Cuban refugees who had served sentences for crimes and were awaiting deportation.

The prison, the oldest in the federal system, now houses more Cubans than any other institution in the nation.

House Judiciary Subcommittee figures show that Cubans make up 5 percent of the total federal prison population but recently were accounting for 10 percent of the federal prison murders, a third of the assaults on staff and half of the assaults on each other.

The 23-acre prison compound and its 17-foot-high walls have held a number of notorious and hardened criminals in the past.

A 1978 survey found the average prisoner to be about 40 years old and doing time for his third serious offense. Overcrowding and other conditions were so bad that violence was a way of life, murders were common and attacks on guards or other inmates almost routine.

There were 16 killings at the Atlanta penitentiary in the late 1970s, before the Cubans arrived, many of them apparent professional ''hits'' related to drug sales.

Weapons, especially hand-made knives, one former inmate testified, were as easy to get as a toothbrush.

William Zambito arrived at the pen in March 1978, and Atlanta prison authorities said they didn't know he had testified in a trial in Miami.

Inmates apparently knew. Zambito was found dead of facial stab wounds 14 hours later.

One victim was doused with turpentine as he slept in his cell and turned into a human torch. Another was hacked to death with a hatchet.

A prison official who testified against the accused killer of Zambito said ''it appears a message was being sent to other prisoners.''

Al Capone was held in Atlanta from 1932 until he was moved to Alcatraz two years later.

Eugene Debs, serving an espionage conviction because of his outspoken opposition to war during World War I, ran the last of his five Socialist presidential campaigns from a cell in Big A in 1920 and got more than 1 million votes. His sentence was commted by President Warren G. Harding in December 1921.

Broadway producer Earl Carroll served a perjury term there after lying about prohibition charges involving a bathtub filled with nude showgirls and champagne.

Soviet spy Rudolph Able was there for five years until he was swapped for U-2 spy plane pilot Francis Gary Powers.

Mafia leader Joe Valachi, who testified against the Cosa Nostra in the early 60s spent time there, as did Vincent Papa, who engineered the theft of heroin from the New York police department, a caper that was the basis for ''The French Connection.''

Papa was found stabbed to death. Later, the two men accused of his killing met similar fates in different prisons.

Since the prison was targeted for closure, it did not get the gradual modernizations other federal prisons got, which meant it was even further behind when the decision was made to bring it up to date.