WASHINGTON (AP) _ The House Appropriations Committee voted Monday to boost spending by the National Endowment for the Arts to $180 million next year and rebuffed attempts to abolish the agency or impose new anti-obscenity curbs on the projects it subsidizes.

The committee, in effect, ducked the politically explosive issue of federal support for ''obscene art'' in favor of House floor action later this week on a separate proposal to shift the responsibility for deciding what is obscene from the NEA to the courts.

An NEA compromise being patched together by Reps. Pat Williams, D-Mont., and E. Thomas Coleman, R-Mo., is tentatively scheduled for floor debate starting Wednesday or Thursday.

The pair circulated the outlines of a proposed compromise among their House colleagues for comment Monday, but Williams said there was no final agreement.

Their proposal reportedly would extend the NEA's life for five years and drop the strict obscenity ban Congress slapped on the agency last fall at the request of Sen. Jesse Helms, R-N.C.

Instead, it would embrace a bipartisan Senate measure that would require the NEA to get its money back from any grant recipient who is convicted of violating obscenity laws. The guilty artist would be barred from receiving any new federal grants for at least three years.

In a gesture to the NEA's congressional critics, the Williams-Coleman compromise reportedly would include a modest increase in the current 20 percent portion of federal funds allocated to state arts agencies.

The bill's new proposed 27.5 percent share for the states is far less than the 60 percent originally proposed by the conservative Coleman, who also had sought a ban on NEA support for any works that violate the Supreme Court's obscenity standards.

The $180 million the Appropriations Committee approved for the beleaguered arts endowment would be a 5 percent increase in its current $171 million budget.

The NEA, an independent federal agency, spends most of the money on grants to artists and arts institutions that must be matched by private funds.

At the urging of Rep. Sidney Yates, D-Ill., a staunch NEA supporter, the committee resoundingly defeated by voice vote a proposal by Rep. Tom DeLay, R- Texas, to eliminate all funds for the endowment until Congress votes to extend its life in separate legislation.

''Some people want to abolish the NEA, and this is what this amendment is all about,'' Rep. Bob Carr, D-Mich., told his colleagues. ''If you are for this amendment, you are for the end of the National Endowment for the Arts,'' Yates said.

Under heavy pressure from Yates and other committee Democrats, Rep. Ralph Regula, R-Ohio, withdrew a proposed amendment that would have banned subsidies for obscene or indecent works and required the NEA to support art that is ''appropriate for a general audience.''

Democrats said Regula's proposal was so vaguely worded it could not be enforced, and might be ruled unconstitutional on the ground that it involved prior restraint on an artist's First Amendment guarantees of freedom of expression.

The committee bowed to pleas by Yates and Rep. Vic Fazio, D-Calif., that it sidestep the ''obscene art'' issue in favor of House action on a Williams- Coleman compromise later this week.

The NEA's proposed $180 million spending budget for fiscal 1991, which began Monday, was included in a $14.5 billion appropriations bill for a variety of federal agencies, including the Interior and Energy departments, the Forest Service and the Smithsonian Institution.

The committee approved the bill by voice vote and sent it to the House floor.