WASHINGTON (AP) _ Governments, colleges and investment firms are scrambling to determine if American companies are complying with anti-apartheid demands in the way they are pulling out of South Africa while still keeping a foothold there.

In a related development, the Transportation Department took steps Tuesday to end air service between the United States and South Africa, in compliance with sanctions enacted into law by Congress over President Reagan's veto.

The department issued an order to revoke landing rights by South African Airways and asked for comments. It said if the order is made final, it will take effect on Nov. 3.

International Business Machines Corp. and General Motors Corp. announced last week that they were selling off their operations in South Africa, apparently partially in response to demands of investors.

But IBM and GM are withdrawing from South Africa in a way that still leaves their products on the market and their plants running, and spokesmen for some institutional investors said Tuesday they are taking a close look to see if such arrangements satisfy divestment demands.

Mark Fabiani, legal counsel to Los Angeles Mayor Tom Bradley, said the city's 3-month-old anti-apartheid law still would prohibit the city from entering into contracts with IBM, GM and other companies that maintain any ties to South Africa.

''It's still possible to support the South African economy through these arrangements,'' Fabiani said of the licensing and franchising arrangements that are leaving some American companies in the South African market despite their withdrawal announcements.

Firms that handle ''socially responsible'' investments also seemed inclined to keep GM and IBM off their list of acceptable investments, said Heidi Soumerai of U.S. Trust Co. in Boston.

Ms. Soumerai, who analyzes such investments for the Calvert Group of mutual funds, said the action by GM and IBM, while laudable, was not enough for the group.

''We try to go beyond direct investment,'' she said, adding there is ''a lot of confusion'' about what U.S. companies are doing in South Africa.

''There is going to be more attention focused on just what is meant by divestment ... what it means varies,'' said Marcy Murningham of the social investment division of Mitchell Investment Management in Cambridge, Mass.

The University of California adopted a divestment policy in July requiring that the university's $3.1 billion investment in companies doing business in South Africa be sold by January 1988.

The actions by IBM and GM seem to satisfy that policy, so the university can continue to hold stocks in that firm, said spokeswoman Linda Fried, even though the situation is ''open for interpretation.''

Roland Machold, director of investment for the State of New Jersey, said the state hasn't made a judgment on whether the two companies are in compliance with state law. Columbia University spokesman Fred Knuebel said he didn't know what the school would do about GM and IBM.

When GM and IBM announced their plans to withdraw from South Africa, they won praise from anti-apartheid activists even though those two corporate giants said they would maintain licensing and franchising agreements with the new owners, former local managers. Those arrangements meant that GM cars and IBM computers could still be sold in South Africa.

Several specialists at colleges and local governments said IBM and GM appear to have satisfied their divestment regulations.

The Investor Responsibility Research Center, which tracks U.S. companies in South Africa, said 116 colleges, 19 states and more than 60 cities and counties have some type of divestment regulation.

The restrictions range from full divestment calling for the sale of stock in companies operating in South Africa to partial divestment banning stock ownership in firms that don't adhere to the Sullivan Principles, a code of conduct for American companies operating in South Africa spelling out hiring and promotion policies for blacks..

IBM and GM, among the 30 American firms which have either left South Africa in 1986 or announced plans to do so, cited the worsening political climate and the white government's failure to eradicate apartheid as reasons for selling.

Despite the corporate action, Richard Knight of the New York-based American Committee on Africa predicted ''these companies are still going to feel a lot of pressure'' from the anti-apartheid movement until they end their involvement totally.

He said the activists' next step in the drive to end apartheid will likely to attempt to cutoff all technology to South Africa, effectively blockading the country.

Ms. Murningham said she expects organizations to re-examine and fine-tune their divestment policies after they see what changes, if any, occur in South Africa in the era of sanctions.

But the announcements by IBM and GM won't affect the Maryland retirement fund because the state's partial divestment law requires only that firms adhere to the Sullivan Principles, according to Howard France, director of investments.