NEW YORK (AP) _ As tradition would have it, the nation's oldest, biggest and most prestigious St. Patrick's Day Parade is lining up again as the most troublesome.

The five-hour, 125,000-strong procession up Fifth Avenue every March 17 has its usual stalwarts: Emerald societies, Gaelic language and music groups, Catholic school students and real-Irish visitors from the old sod.

But there's no room on the guest list for the Irish Lesbian and Gay Organization, which wants to march under a prominent banner and seeks backing from the courts.

The Ancient Order of Hibernians, which has sponsored the New York parade since 1838, says it's sticking by its rules.

''The parade is dedicated to Saint Patrick, with emphasis on the saint,'' said spokesman Frank Feighery. ''The only political view or slogan allowed in the Saint Patrick's Day parade is 'England Get Out Of Ireland.'''

On Friday, a judge from the city's Human Rights Commission will consider whether the Hibernians are discriminating illegally in banning a distinctly- identified homosexual contingent.

And on March 12, a federal judge will hear arguments on the contingent's attempt to have the parade banned if it is excluded.

Last year, the group's members and supporters marched as individuals but were prohibited from carrying an identifying banner. Mayor David Dinkins passed up the mayor's traditional spot in the parade to march with the homosexuals. Spectators jeered and threw beer cans at him.

This year, the Hibernians banned the gay group's marchers because of last year's ''outrageous conduct,'' including alleged obscene gestures in front of St. Patrick's Cathedral.

The New York Civil Liberties Union said in papers filed Wednesday with the Human Rights Commission that the city has no right to force the Hibernians to let homosexuals march.

''We're balancing two conflicting civil liberties interests,'' said Rob Levy, NYCLU senior staff attorney. ''But ultimately the First Amendment prevails.''

''Could the organizers of the Israel Day parade be compelled to accept German-born neo-Nazis to its ranks?'' Levy asked.

The police department's Emerald Society has advocated banning gay marchers, while Grand Marshal Cornelius Doolan was reported to be lobbying to allow them. Some advertisers dropped out of the parade's nationally syndicated broadcast over the controversy.

The Hibernians complained that some marchers last year wore T-shirts with homosexual slogans, kissed and made obscene gestures when onlookers taunted them.

''There isn't any evidence of our 'outrageous behavior,''' said Anne McGuire, an Irish Lesbian and Gay Organization spokeswoman. ''The outrageous behavior was on the part of the Hibernians who turned their backs, and the spectators who chose to scream at us and throw beer cans. We think it's just another excuse to exclude us.''

City Council Member Tom Duane, a gay Irish-American, said he hadn't experienced ''such palpable hatred since I was attacked by a group of harassers outside a gay bar on Long Island several years ago.''

Feighery said the parade committee has chosen some 200 groups, including Hibernians from the region, visitors from all 32 counties of Ireland and an assortment of groups promoting culture.

''It's an Irish Catholic parade; that's the criteria,'' Feighery said, adding that some non-Irish groups are allowed to march if they express ''some affinity'' with the criteria.

Anti-abortion banners are banned. ''If we don't even proselytize our own agenda, why should we allow someone whose agenda is opposed by the Catholic church?'' Feighery said.

Some Hibernian officials would like concessions if the gay group is allowed to march: conservative clothes, no blatant banners, no marchers from other gay groups. They also insist all new participants march at the end of the line. The 1959 Hibernian rule book states: ''Seniority cannot be trifled with.''

Several politicians, including the city's comptroller and council president, said last week they won't march unless the gay group's members can participate ''in a manner which they deem acceptable.''

Dinkins said: ''I don't see it as useful to talk about what I'm going to do 'if,' ... We're going to have a Saint Patrick's Day parade that's going to be inclusive and I'm going to keep working at that.''