NEW YORK (AP) _ ABC has notified A.C. Nielsen Co. that it wants to end its agreement with the ratings company effective Aug. 31, just before the beginning of the new television season, according to a report published today.

The Daily News reported that ABC, the third-ranked network, was dropping Nielsen just as the ratings service was set to change its way of measuring a show's popularity.

The company uses an audiometer, which is attached to the back of television sets, but soon will install People Meters, a system in which viewers push buttons to record what they watch.

CBS said it did not object to the system, but NBC and ABC had said it should be given more study.

The networks pay Nielsen about $3.5 million annually for their daily ratings service. The ratings are used by networks and stations to determine how much they charge for advertising.

Marvin Mord, ABC vice president of marketing and research services, said ABC was terminating the agreement because it wanted to ''explore more fully the range of options available for the '87-'88 season.''

The newsaper reported that the three networks were considering picking up a new service, Audits of Great Britain.

Mord said the network might return to Nielsen.