WASHINGTON (AP) _ Faced with billions of dollars in defaults on student loans, the government said Tuesday it is resorting to ''the ultimate trump card'' to collect - the Internal Revenue Service.

Education Secretary William J. Bennett said his department is asking the IRS to withhold tax refunds for 1 million defaulters on federal student loans unless they start paying their debts.

Another 1 million borrowers will get notices from state agencies warning that they will be denied federal tax refunds next year unless they make good on their debts. The states are acting on loans made by state agencies and insured by the federal government.

''I think it's going to be without a doubt the most successful thing we've ever done to recover defaulted loans,'' said Richard Hastings, director of debt collection for the department.

''We're talking probably hundreds of millions'' of dollars, he said.

Department officials say former undergraduate and graduate students have defaulted on $3 billion in low-interest loans subsidized by the federal government under the Guaranteed Student Loan program. Another $1.1 billion has been defaulted in the National Direct Student Loan program of low-interest loans to students through their schools.

All those in default are now out of school, and while some may not be working, most are earning money and ''basically are making economic decisions'' not to repay the low-interest loans, Hastings said.

''Now we're playing the ultimate trump card, which is to use the IRS,'' he said.

The move was welcomed by the National Association of Student Financial Aid Administrators, representing 3,100 schools that handle federal aid to students.

Dallas Martin, executive director of the association, said 5 percent to 8 percent of recipients default, but that the public believes the figure is much greater.

''We hope that the people that aren't being responsible will realize they've got to repay,'' he said. ''I think it'll change the public perception a little bit.''

Bennett said in a statement that notices will be mailed to those in default beginning this Saturday. Defaulters will have 60 days to begin repayment or to work out a plan for payment.

If no arrangements are made, the department will have the IRS withhold any federal income tax refund due that person, up to the total owed, beginning with the 1985 tax year. The money would go to the federal treasury, thus wiping the loan off the department's books.

''This is a major step which should show loan defaulters that we are dead serious about collecting these debts to American taxpayers,'' Bennett said.

The department is acting under authority granted by Congress last year. The action followed a pilot program by the Department of Health and Human Services withholding federal tax refunds from people with delinquent child support payments.

Similar programs are planned to collect debts owed the Veterans Administration, the Small Business Administration, and the departments of Agriculture and Housing and Urban Development, said IRS spokesman Steve Pyrek.

The IRS so far this year has offset 1984 tax refunds to 465,000 people, amounting to $221 million, for overdue child support, Pyrek said.

The Education Department has used private collection agencies against defaulters and recently acted to blacken the credit ratings of student loan defaulters by turning their names over to national consumer credit bureaus. It also has referred the names of 16,000 defaulters to the Justice Department for prosecution.