NEW YORK (AP) _ International Business Machines Corp. says it expects to report a decline in its worldwide orders and shipments next week, a piece of bad news that took the spotlight off its introduction of four mid-size computers.

IBM's stock dropped $5.37 1/2 a share Tuesday to $128.12 1/2 in consolidated New York Stock Exchange trading, reflecting nervousness over prospects for its earnings report next week.

IBM's decline prevented a rise in the Dow Jones average of 30 industrial stocks, which closed unchanged at 1,784.45.

''Our business outside the U.S., while still growing, is beginning to moderate,'' IBM spokeswoman Pamela Hawkins said in a statement in response to questions from reporters about the stock decline.

Although its U.S. business is stable, IBM expects its worldwide orders and shipments for the first nine months of 1986 to fall below the same period in 1985, and pressures on the company's margins to continue, she said.

Most analysts expect IBM to report a decline in profits for 1986, marking the first time that has happened two consecutive years since the Depression.

The new 9370 computers are designed to combat depredations on IBM's mid- size machines by minicomputer makers, mainly Digital Equipment Corp., analysts said.

''I don't think it's going to kill DEC, but it could make IBM more competitive,'' analyst Steve Milunovich of First Boston Corp. said of the new line. ''It could have more effect on the secondary players like Data General,'' he added.

Digital, the Maynard, Mass.-based computer maker, has capitalized on the lack of compatibility between IBM computers. In spite of Tuesday's announcement, ''We have still a very formidable challenge for them and a real alternative for our customers and their customers,'' said William Johnson, the company's vice president for distributed systems.

The 9370 computers belong to the same System 370 family as IBM's successful line of big, powerful mainframe computers and run many of the same programs but are much smaller and cheaper. The smallest, the Model 20, is a bit larger than a two-drawer filing cabinet and starts at $31,000 for the processor alone. Operating software and memory devices are extra.

Pilot installations are scheduled to begin in the second quarter of 1987, and IBM said it did not expect to begin volume shipments until the second half of the year. However, Milunovich speculated that the actual delivery date might be moved up.

IBM said it had no plans to withdraw or lower the prices of the smallest members of its 4300 family of mainframes, even though the new computers are faster and cheaper.

IBM also announced it was changing the pricing policy for its mainframe software so it would be less expensive for smaller mainframes such as the new 9370s and 4300s than for the biggest mainframe computers, the 3090s.

IBM said that the new computers, along with the enhancements to its System 36 and System 38 departmental computers announced in June, ''add up to the most comprehensive mid-range offerings available today.''

Ms. Hawkins, the IBM spokeswoman, said she did not know precisely why IBM's stock began to fall before the comments about orders and shipments were conveyed to reporters. But she said financial analysts may have found out the news through their regular contacts with the company's officials in investor relations.