NEW YORK (AP) _ For generations the unwritten code of conduct among bankers rendered hostile takeovers virtually taboo; that is, until the oldest bank in the country saw fit to break with tradition.

Bank of New York Co. - founded by Alexander Hamilton eight years after the the Declaration of Independence was signed and so ingrained in tradition it was sometimes dubbed the sleeping giant - startled its colleagues by making a brash move for control of rival Irving Bank Corp. last fall.

The joke around Bank of New York and elsewhere had been that before Hamilton was killed in a duel, he told his employees not to do anything until he returned, and that was the way business has been conducted ever since.

''That's the impression a lot of people on the outside looking in have had of the Bank of New York,'' said company spokesman Owen Brady. ''Of course, that's not true.''

Under the leadership of J. Carter Bacot since 1982, Bank of New York has moved to expand its market. Its twice-revised tender offer for Irving of $1.08 billion began last month with regulatory approval but without Irving's blessings.

Although it wasn't the first hostile bank bid, it is the largest deal of its kind to have progressed this far in a close-knit industry that tries to avoid controversy.

Banking analysts say hostile bids could become more commonplace, particularly among aggressive regional institutions, although they don't foresee the same type of takeover frenzy prevalent in other industries since the Oct. 19 stock market crash made share prices attractive.

''The trend is evolutionary, not revolutionary,'' said James J. McDermott Jr., an analyst with Keefe Bruyette & Woods. ''However, we do think that the ground has been broken with Bank of New York and that under the appropriate financial circumstances we could see other hostile offers.''

The Federal Reserve's approval of the Bank of New York bid - in which the Fed declined to make any distinction between a friendly or hostile offer - may encourage other institutions to take chances on takeovers even under stormy conditions.

Banks have been merging, under friendly terms, at a fierce clip in recent years to keep pace with economic conditions, interstate banking law and new types of non-bank competitors like Sears, Roebuck & Co. The industry also has been struggling with deteriorating profits due to shaky loans to Third World countries and to depressed domestic oil and farm regions.

Since 1982, banks have led the nation in mergers and acquisitions, with an average 260 friendly deals a year during that period. Many more are expected as more states open their doors to outsiders.

Despite the motivations, the nation's 14,000 commercial banks face significant hurdles in the merger process that other industries don't have to clear.

For one thing, banks have to go through a laundry list of local and government regulations, not to mention waiting periods and certain capital requirements, before even trying to win over shareholders.

Among the agencies to pass judgment are the Fed, state banking boards, Securities and Exchange Commission, and in some cases local community groups. And depending on the bank's charter, the comptroller of the currency also could step in.

''It's like a 12-course meal. It's very easy to overcook and undercook at any given time, and getting them off the table at the right times is not easy,'' said Michael Rosen, a Boston attorney specializing in bank mergers.

''Banking is really, in a way, quasi-public. It's different from other businesses. It (banks' coexistence) is definitely more than a gentlemen's agreement.''

Unlike other businesses, which, for the most part, can continue selling or manufacturing uninterrupted, banks have to worry about maintaining their trademark personal service.

''The banking business ... is really a people business,'' explained Stephen Berman, an analyst with County Securities USA. ''As a rule a bank has gotten to the position it's in, based on its people and management.''

The depature of certain managers after a hostile takeover, for instance, could jeopardize a loan agreement that might have been in the works or force wary customers to move their deposits elsewhere, he said.

Berman also noted that hostile suitors, in particular, who are not privy to all of a target bank's books face additional obstacles.

''A bank's assets are much more difficult to judge than an industrial company,'' he said. '' ... The principal asset of a banking company is that loan portfolio composed of thousands of individual loans. When you get down to it we know little about them.''

In the proposed Bank of New York-Irving merger, many analysts agree that a combination of the two wouldn't be such a bad idea since both New York banks operate in a similar marketplace and offer similar services like securities processing.

''More to the point Irving has been an underperforming institution,'' said McDermott.

But most analysts are split as to whether Bank of New York will succeed in creating one of the nation's 15th largest bank holding companies with assets of more than $40 billion.

Bank of New York's current offer - of $15 in cash and 1.575 shares of its own stock for each of Irving's roughly 18.1 million common shares outstanding - has been denounced as inadequate by Irving's board of directors.

Irving, which was founded in 1851 and also criticized as being a ''sleeper,'' has stressed its desire to remain independent, although it said it was talking with other parties and may seek a ''white knight,'' or friendly merger partner as an alternative to Bank of New York.

It has taken a number of steps to spurn Bank of New York's advances, including instituting an anti-takeover ''poison pill,'' intended to make any hostile buyout prohibitively expensive, and filing suit to block a proxy solicitation by Bank of New York. Shareholders are scheduled to vote on Bank of New York's proposed slate of 16 nominees to Irving's board in about two weeks.

Meanwhile, industry watchers say that if Bank of New York succeeds it will be especially noteworthy, given the track record of similar proposals.

Two years ago, First Interstate Bancorp made an unsolicited $3.2 billion bid for ailing BankAmerica Corp., but the proposal was spurned and the offer withdrawn. Last year, Banc One Corp. became Marine Corp.'s white knight after Marshall & Ilsley Corp. made a $560 million cash and stock offer.

In fact, since 1981, there have only been three significant successful hostile deals, although all have been under $100 million, according to Mergers & Acquisitions magazine.

''All bankers know each other. They all go to the same social events. It's not easy to be an outcast,'' said Rosen.

End Adv Sunday April 10