ORLANDO, Fla. (AP) _ Harcourt Brace Jovanovich Inc. asked a judge Monday to clarify the rights of debenture holders in a squabble that arose in connection with the company's efforts to avoid a hostile takeover.

Attorneys for the Orlando-based publishing and entertainment concern told Orange County Circuit Judge Fredrick Pfeiffer that debenture holders who failed to convert their bonds by a June 8 deadline should lose their right to trade them for stock.

If the judge holds that they still have conversion rights despite expiration of the conversion deadline, Harcourt asked that the debentures be traded in for stock under a formula that would allow for fewer shares.

At the hearing were attorneys for Harcourt, and the trustee for its debentures, Sun Bank. Also represented were British Printing & Communication Corp. and the investment firm Salomon Bros. Inc. which account for most of the $44 million in debentures that were not traded for stock.

An attorney for British Printing said the legal issue turned on whether the contract right of a debenture holder ''could be extinguished by HBJ.''

The London-based British Printing, controlled by its chairman, Robert Maxwell, proposed last month to buy HBJ for $44 a share, or about $1.73 billion.

J. William Brandner, a Harcourt executive vice president, said the board of directors considered Maxwell's offer financially inadequate.

The HBJ board rejected Maxwell's offer and proposed a $3 billion recapitalization plan that included payment of a special dividend of $40 a share in cash and a fraction of new preferred stock worth $10 a share.

HBJ had $200 million in convertible debentures outstanding and the publishing concern gave the shareholders until June 8 - the record date for the special dividend - to convert the debt into common stock.

It said $156 million were converted. Of the remaining $44 million of the debentures, $22 million were held by Salomon and $9.3 million by British Printing.

The price for converting those debentures into HBJ stock was fixed at $34 a share. Under the formula using a $34 conversion price, each $1,000 bond would produce 29.41 shares of stock per bond.

Sun Bank has argued that the rules governing the debentures would allow the debt holders to convert to common stock at a price of $1 a share after June 8.

Under that reasoning, Salomon would be able to convert their debentures into 22 million common shares, and the other remaining debenture holders would be able to convert their debentures into an additional 22 million shares.

Harcourt currently has about 39.4 million common shares outstanding. If all remaining debenture holders converted, Harcourt would have 83.4 million shares and the debenture holders would own a controlling 53 percent of the company.

Sun Bank attorney Robert Myers told Pfeiffer that it is not the function of the court to rewrite a contract.

Darryl Bloodworth, attorney for British Printing, asked the judge to declare a conversion price of $1 a share.