NEW YORK (AP) _ The biggest is getting bigger with an agreement by Computer Associates International Inc. to buy troubled Cullinet Software Inc. for about $320 million in stock.

Computer Associates is the world's first independent software company to achieve $1 billion in annual revenue, while Cullinet has fallen on hard times and lost money each year since 1986.

The companies announced a definitive merger agreement Monday, subject to shareholder and regulatory approval. Computer Associates, of Garden City, N.Y., said it expected to close the deal in August.

Analysts said they expect Computer Associates to follow its tradition of slashing jobs at new acquisitions, a possibly painful prospect for John Cullinane, the founder, chairman and biggest stockholder of Cullinet, which is based in Westwood, Mass.

''Cullinane is paternal. It was clearly not the first choice of John Cullinane that Computer Associates be the company that acquired it,'' said Charles Federman, a partner of Broadview Associates, an investment banking firm in Fort Lee, N.J.

Computer Associates and Cullinet compete with each other and with International Business Machines Corp. in selling software for big mainframe computers made by IBM. Computer Associates also sells software for smaller computers.

Computer Associates is headed by Charles B. Wang, an aggressive executive who has built the company from an also-ran mainly by acquisitions and has promised to keep the company growing at 30 percent a year.

Last year his company had revenue of $1.03 billion - more than better-known software makers such as Microsoft Corp. and Lotus Development Corp. - and a profit of $163.5 million.

''The proposed merger will combine the talents, technology and resources of our two companies for the benefit of our clients, employees and shareholders,'' Wang said in a statement.

John Cullinane founded Cullinet in 1968. He retired in September 1987 but came back in March 1988 as the company continued to falter. The company said Monday it lost $19.6 million last year on revenue of $202.9 million.

Computer Associates acquisitions have included Applied Data Research Inc. for $170 million last year and Uccel Corp. for $800 million in 1987.

Cullinet, itself once the world's biggest independent software company, began to stumble when IBM targeted the mainframe data-base software business in 1985.

A Justice Department spokesman said the department would have no immediate comment on the proposed transaction.

The possibility that the Justice Department might block the acquisition on antitrust grounds apparently kept some speculators from buying Cullinet stock, Federman said. The stock sold Monday at a discount to the value implied by the Computer Associates offer.

Computer Associates shares fell 75 cents to close at $19.50, while Cullinet rose 75 cents a share to $9 in consolidated New York Stock Exchange trading. Computer Associates issued a 2-for-1 stock split Friday.

Shareholders of Cullinet will get half a share of Computer Associates stock for each share of Cullinet they own. Based on Monday's closing price for Computer Associates, that amounts to $9.75 for each of the 33 million Cullinet shares outstanding.

The transaction will be accounted for as a tax-free pooling of interests.

Computer Associates approached Cullinet about a merger last year but was rebuffed. Earlier this year, Cullinet hired Goldman, Sachs & Co. to shop the company around, but apparently received no offers better than the one from Computer Associates.