TALLADEGA, Ala. (AP) _ Many of the 15 inmates of the federal prison near Talladega who are slated for return to Cuba say they worry about bad times and possible reprisals on the island they fled.

''All you need to know about Cuba you can learn by looking at the number of Cubans who have come here,'' said Mario Sosa Hernandez, who has been in prison since being caught in San Francisco in 1984 with $25 worth of heroin he insists wasn't his. ''I could get hurt a lot if I have to go back there.''

Sosa said his record in Cuba was clean before Fidel Castro's communist takeover in 1959. ''After the victory of the revolution I had a problem with some violence, and with marijuana,'' he mused, his hands cuffed behind his back in a tiny locked interview room.

Sosa, 50, and the other 14 were among the 125,000 Cubans who arrived on the five-month ''Freedom Flotilla'' boatlift from the Cuban port of Mariel in 1980.

In the wake of U.S. charges that Castro had used the boatlift as an opportunity to rid Cuba of some prison and mental hospital inmates, the Cuban leader agreed in 1984 to take 2,700 of the boatlift people back. But after taking only a fraction of that number, Castro rescinded the agreement in 1985.

When Cuba announced a year ago that it would resume the agreement, Cubans fearing deportation grabbed guards as hostages at federal prisons in Atlanta and Oakdale, La., holding the prisons for nearly two weeks.

The 15 inmates in prison here are among 114 who are in Immigration and Naturalization Service custody because of past crimes and have been denied parole. Many others among the original list of 2,700 are either paroled or are in state custody.

The U.S. Justice Department so far has reviewed 28 of the 114 cases and decided on deportation for the 15.

Federal courts have ruled that Cubans classified as ''excludable aliens'' can be held indefinitely until they are deemed ready for parole or are accepted by Cuba or another country.

Lawyers for the Cubans are seeking an administrative review, and the government has delayed deportations until the issue is decided.

Meanwhile, the 15 wait with 99 countrymen at Talladega, two to an 85- square-foot cell with five hours of exercise and three showers a week. When they leave the cells, they are in cuffs and chains.

They say it beats Cuban jails. The Justice Department panels consider, among other things, the inmate's behavior in Cuba and in American prisons, his potential for violence, job prospects and personal harm that might follow a return to Cuba.

Renato Oliva-Sanchez, arrested for selling cocaine in 1983, said he had a clean record in Cuba until he got in trouble with the army for refusing to serve in a special services unit.

Oliva, who had a string of lesser arrests before his last one, said his contact with family members here and in Cuba has been ''very sporadic.''

Most of his family, he said, is in the Cuban military or otherwise close to the government. ''When I write, I have to send the letters to another family and they give them to my family. I can't write directly.''

The United States sent 201 Cubans back before Cuba suspended the 1984 repatriation agreement in 1985. Of the 201, about 156 are free after spending six weeks in quarantine. A few have died. Those with time left on their American sentences are serving it.