---
title: "The agent will make the check pass"
description: "Given a red signal, an agent optimises for the signal. That is not a flaw in any particular tool — it is what optimisation means, and it changes which signals are worth having."
url: https://harpywatch.com/blog/the-agent-that-fixed-the-symptom
kind: article
author: "The HarpyWatch team"
published: 2026-09-03T09:00:00+00:00
tags: ["AI Agents", "Monitoring", "Reliability"]
publisher: "HarpyWatch"
---

# The agent will make the check pass

A test fails intermittently. You hand it to an agent and say: make this stop
failing.

There are two things it could do. It could work out that the test starts a
server and asserts against it before the port is listening, and make the
assertion wait on the actual readiness condition. Or it could wrap the
assertion in a retry with a short sleep.

Both make the test green. One takes forty minutes of reading and a guess that
might be wrong. The other takes nine seconds and almost always works, because a
retry loop converts an intermittent failure into an intermittent delay.

Only one was asked for. Both satisfy the request as stated.

## This is not a complaint about agents

You can file this as a quality problem — better models read more carefully,
better prompts say "fix the root cause" — and get some of the way. The
structure underneath does not move: **when you give something a signal and ask it to turn
the signal green, you have specified the signal, not the property you wanted.**
An optimiser takes the cheapest available route to a satisfied condition. If a
route exists that does not involve fixing anything, it is a correct solution to
the problem you actually posed.

Humans do this too, and we have a whole vocabulary for it — teaching to the
test, gaming the metric, Goodhart's law. What is different now is throughput and
traceability. When a person suppresses a warning to get a build through, there
is someone who knows they did it. When forty files acquire the same treatment in
an afternoon there is no such record, and each line reads as reasonable.

## The shape of a symptom fix

The pattern is easy to recognise once you have the category, and hard to see one
commit at a time. The same move, five ways:

| The complaint | The cheap green | What went unfixed |
| --- | --- | --- |
| This page is slow | Add a long `Cache-Control` max-age | The query behind it |
| This test is flaky | Retry three times | The race |
| This request fails sometimes | Raise the timeout | Whatever takes that long |
| Type error at this line | Widen the type, or cast | The value that should not be there |
| This check is noisy | Loosen the threshold | The thing crossing it |

Every one of these is a legitimate engineering action in some context — a cache
header is often exactly right, a timeout is sometimes genuinely too tight. That
is what makes the category hard to police: the symptom fix is not a wrong
action, it is the right action applied for the wrong reason, and the reason is
not in the diff.

The cache header case is worth sitting with, because the damage is delayed and
lands somewhere else. A slow page gets a max-age. The page is now fast for
anyone who receives a cached copy, the measurement improves, and the report says
the work is done. The query is still slow — for the first visitor after every
deploy, for every authenticated request that cannot be cached, and for the
database, which now does the same expensive work in bursts after each deploy and
each eviction: lower on average, spikier, and on nobody's graph. And a second
failure mode has been added, one that did not exist before: a document cached
longer than it should be, serving last week's content after this week's
deploy.

## The property that distinguishes a good signal

If the cheapest satisfying route is the one that gets taken, the useful
question is not "how do I stop it doing that". It is **what does the cheapest
route cost when the honest one is the only route available**.

Signals differ enormously in that.

An assertion inside your own repository is the weakest case. The thing measured
and the thing measuring live in the same tree, under the same permissions, and
whatever must satisfy the check can also edit the check. Deleting the assertion,
weakening the fixture, adding `skip` — all one-line fixes, none of which stand
out in a diff of two hundred lines. Such a signal is only as strong as the
attention of whoever reads that diff, and attention is what runs short first.

A signal measured from outside is a different kind of object. If the statement
is "a GET to `https://example.com/pricing` from a machine you do not control
returns a 200 with the expected content, a valid certificate chain, and a first
byte inside a budget", then there is no edit that makes it true except making
it true. You cannot cache your way past a redirect loop. You cannot retry your
way past a missing intermediate certificate — the client either builds a path
to a trusted root or it does not. No cast, threshold or annotation makes a
sitemap's URLs return 200 when the pages behind them have been deleted.

The boundary matters, because the cache example above crosses it. A
`Cache-Control` header does make an external latency measurement faster without
the query getting faster, and that is a real limit on this argument. What
resists it are the checks whose subject is a *fact* rather than a *budget*: what
the chain contains, where a redirect terminates, whether a sitemap's URLs return
200, whether the rendered page contains the string it should. No header changes
those. Thresholded signals can be moved by configuration; factual ones cannot,
and that is which to rely on.

## What this asks of you, practically

Three things follow, and none of them require distrusting the agent.

**Keep at least one signal outside the blast radius of the change.** If every
check that reports on your site can be edited by the same session that edited
the site, you have one system reporting on itself. It is the same argument as
not running monitoring on the servers it monitors, moved one layer up: there the
compromise is availability, here it is authorship.

**Prefer checks that assert on what a visitor gets.** Response status, rendered
content, headers on the wire, the certificate actually presented, the redirect
chain a browser follows. These are facts about the deployed system rather than
the repository, and they stay true statements regardless of who wrote the code
or how carefully anyone read it.

**Look at the deployed thing before the traffic does.** None of this has to
wait for an incident. Point the same external checks at a staging hostname and
read the result before you move DNS. The redirect that loops, the header that
vanished, the page that renders a stack trace — all visible to an outside
request the moment the thing is reachable.

## The honest limit

An external check cannot tell you that the retry was the wrong fix. It never
looks at your repository, so a suppressed test and a passing one are the same
thing to it. If an agent papers over a race condition, HarpyWatch will not
notice, and neither will anything else that observes from outside — right up until the race produces a
response a visitor can see, which is exactly the moment it becomes our problem
rather than yours.

So this is not a replacement for reading the diff. It is the part of
verification that does not depend on anyone having read it — and as the volume
of generated change rises, the share of it genuinely reviewed falls.

Prefer the checks whose only cheap route runs through the actual repair. They
do not remove the need for someone careful — the piece above is a list of the
gaps they leave — but they shrink the set of problems that can be closed
without being solved, and that set is where the expensive surprises live.