BOSTON (AP) _ Gov. Michael S. Dukakis and other state leaders say the state does not have an estimated $400 million needed to comply with a judge's order of an immediate 30 percent increase in welfare benefits.

They also said Monday that, under the Massachusetts Constitution, Superior Court Judge Charles Grabau lacks authority to order another branch of government to spend public funds.

''I'm afraid I just don't agree with him,'' Dukakis said shortly after Grabau ruled in favor of three homeless mothers who sued the state last year.

Grabau ordered Welfare Commissioner Charles Atkins to boost benefits for a family of three from the current $476 a month to as much as $926 a month.

The judge also ordered the administration to stop, within 30 days, placing homeless welfare families in emergency shelters, hotels and motels for periods of more than 90 days.

''As this court's order has been met with indifference or misunderstanding, this court is compelled to spell out more explicitly the steps to be taken in order to achieve obedience to the judicial mandate.''

Last June, Grabau found Atkins was violating a 1913 state law that requires annual calculations of a level of assistance high enough to allow welfare recipients to raise their children in their own homes.

Dukakis said he would ask the attorney general to appeal the ruling on grounds that it violated the doctrine of separation of powers.

It was unclear immediately how Dukakis could increase benefits without an appropriation by the Legislature to provide the state's 50 percent share of the cost.

Human Services Secretary Philip Johnston said compliance with Grabau's order could cost $400 million.

''We certainly don't have $400 million in the bank right now, and I don't know how we would possibly come up with it,'' Johnston said.

He defended the Dukakis administration, saying it had raised welfare benefits by 32 percent over the past four years,

Sen. Patricia McGovern, who chairs the powerful Senate Ways and Means Committee, agreed that the state could not afford a major increase in welfare benefits.

''It's an enormous amount of money, an incredible amount,'' said McGovern. ''It would mean a major tax increase, and I think there is not a will to do that.''

Katherine Mainzer, director of the Massachusetts Coalition for the Homeless, which joined in the suit, said she was confident the ruling would be upheld. ''Today, homeless welfare families are celebrating.''

About 83,000 other families are receiving Aid to Families with Dependent Children, the principal welfare program.