WASHINGTON (AP) _ Former Housing and Urban Development Secretary Samuel Pierce, who refused to answer questions at a congressional hearing last month, said Thursday he will invoke his constitutional rights and maintain his silence at sessions scheduled for Friday and next week.

Pierce urged the panel to cancel plans for Friday's scheduled hearing, but he was swiftly rebuffed by the subcommittee investigating scandals at his department during the Reagan administration.

In another development, Pierce's attorneys accused a key witness against the former secretary of important inconsistencies in her account to the subcommittee of how a housing project received funds.

''Mr Pierce is concerned that there continues to be an atmosphere wherein he remains a target of the subcommittee's hearings,'' his attorneys wrote to Rep. Thomas Lantos, D-Calif., chairman of the housing and employment subcommittee of the Government Operations Committee.

They added, ''It is our hope that some time in the near future the present level of tension may be reduced such that Mr. Pierce can reconsider'' his decision not to answer questions.

The refusal by Pierce and three other former top HUD officials to testify has left the 6-month-old congressional investigation at a crossroads. The panel has heard from dozens of developers, consultants and attorneys who did business with the department about mismanagement and political favoritism during Pierce's tenure, but has received little testimony from high agency officials from the period.

Lantos has raised the possibility of offering immunity to two of the witnesses who refused to testify, Deborah Gore Dean and R. Hunter Cushing, a former Dean aide who later became deputy assistant secretary for housing.

But several subcommittee members oppose any immunity grants and some have suggested dropping the inquiry and leaving the remaining investigative work to law enforcement agencies and subcommittee staff. Federal investigators are looking into hundreds of cases.

''We've accomplished 90 percent of what we needed to accomplish,'' said Republican Rep. Christopher Shays of Connecticut. ''Now we need to take what we've learned and make changes at HUD.''

Pierce asked that hearings scheduled for Friday and Nov. 3 be canceled and subpoenas demanding his presence vacated. Forcing him to appear when it was clear he would not testify would subject Pierce to unnecessary adverse publicity, ''personal embarrassment and is punitive in nature,'' his attorneys said in the letter.

A committee aide, however, said there were no plans to cancel the Friday session and a decision would be made later on whether to hold next week's. Several members said they would vote to drop the subpoena for next week. ''I don't want to go through a charade,'' Shays said.

Congressional sources, speaking on condition of anonymity, said Lantos has told Pierce attorneys Paul L. Perito and Robert Plotkin that he believes there is legal precedent for the subcommittee to compel Pierce to testify, since he appeared at a hearing in May and answered questions. But these sources said Lantos has no plans to force the issue.

Pierce based his refusal on the constitutional provision, found in the Fifth Amendment, that no one may be compelled to give evidence that may tend to incriminate him.

Pierce's attorneys, meanwhile, released documents they obtained from HUD and said the papers raise questions about the testimony of a former HUD official, Shirley McVay Wiseman. She has testified that she was ordered by Pierce to release money for a North Carolina housing project, a claim that has prompted several lawmakers to accuse Pierce of perjury because he had previously testified he never personally ordered any individual project to be financed.

Ms. Wiseman told Congress in July that when she refused to approve the project she was called by Pierce. ''I want the project funded,'' she quoted the former secretary as saying.

The documents, however, show that Ms. Wiseman told department investigators that Pierce's top aide, Ms. Dean, called her and ordered the project funded.

The documents were from an internal investigation of the project, and Wiseman, according to the notes of the investigator who interviewed her, made no mention of a call from Pierce even when asked about rumors that political favoritism played a part in the approval.

Wiseman, according to a memo from the investigator sent to Pierce's attorneys, ''did not tell me, in words of substance, that Secretary Pierce telephoned her and told her, 'I want the project funded.'''

According to the investigator's 1985 notes, Ms. Wiseman said that after she refused to sign the approval papers ''she was directed by the 10th floor, Deborah Dean, to fund the project. ... Dean, she believes with the Secretary Pierce's concurrence, directed Wiseman's staff to write the waivers for the project and to fund it.''

Wiseman, now president of the National Association of Home Builders, did not return a telephone message left at her office Thursday.

''It would seem logical to me that if Pierce had called her she would have said 'he called me' and not that she thought Dean had the secretary's concurrence,'' Perito said. ''If she didn't mention the call from the secretary months after the project was funded, it seems strange that she can remember it so clearly years later.''