---
title: "Verify the artefact, not the diff"
description: "Four agents changing one codebase means four diffs nobody has time to read end to end. The answer is not fewer agents: it is checking the deployed site, which costs the same however many changes arrived."
url: https://harpywatch.com/blog/verify-the-artefact-not-the-diff
kind: article
author: "The HarpyWatch team"
published: 2026-10-15T09:00:00+00:00
tags: ["AI Agents", "Deployment", "Reliability"]
publisher: "HarpyWatch"
---

# Verify the artefact, not the diff

Four agents start work on the same repository at nine in the morning. One is
restructuring the navigation, one is adding a checkout step, one is upgrading a
dependency, one is rewriting the marketing copy. By lunchtime every branch is
green and merged. Nobody read all four diffs end to end. Not because the team is careless: careful review is slow work, and four
substantial diffs is enough of it to consume a day that already had a purpose.

The instinct is to slow down: fewer agents, smaller changes, mandatory review
before merge. That instinct is aimed at the wrong quantity. Review spends human
attention, and it spends it per change: four merges means four diffs to read.
Checking the deployed site spends requests, and it spends them per URL — the
same list, whether one change landed today or forty. Adding a fifth agent adds
a fifth diff. It does not add a URL.

## What the diff cannot show you by construction

A diff describes a transformation: this file, these lines, before and after.
What a visitor receives is a property of the whole deployed tree, the
configuration around it and the data underneath it. Most of what breaks a site
under concurrent change lives in that gap — in the result, not in any of the
transformations that produced it.

**Two correct changes that are wrong together.** Agent A renames `/pricing` to
`/plans` and updates every reference it can find. Agent B, in a branch that
started an hour earlier, writes a new footer that links to `/pricing`. Both
diffs are internally consistent. Both pass review, if anyone reads them. Git
merges them without a conflict, because they touch different files. The 404
exists only in the merged result, which is the one artefact no reviewer was ever shown.

**A rename that outruns the things that reference it.** The same problem
without a second agent: a route changes and the references in a database row, a
seeded fixture, an email template, or a Markdown file in a content directory do
not move with it. A search-and-replace across source code sees the code. The
references that live in data are invisible to the diff by definition.

**A dependency upgrade that changes runtime behaviour and not one line of your
code.** The lock file moved. Nothing else did. A reviewer reading the diff sees
a version number. The change is in what the library does when it renders,
serialises or redirects, and it is only observable by running it.

**Configuration that was correct on the branch it was written on.** An
environment variable, a base path, a cache header set for the branch preview
and not the production host. The diff is right. The composition of that diff
with the deployment it lands in is not.

Every one of these failures is *between* changes rather than *inside* one.
Reviewing each change more carefully does not find them, because none of the
changes is individually wrong. A reviewer catches this class only by holding two
branches in mind at once and noticing the interaction — which happens, and does
not happen reliably, and does not happen at all once the branches outnumber the
reviewers.

## Why "slow the agents down" is an incomplete answer

Throttling does reduce the number of unreviewed states, and so do branch
protection, staged rollouts and a test suite that gates the merge. All of them are worth having. What none of them does is look at the merged,
deployed result: the 404 above exists only there, and a rule about how changes
get merged cannot observe it.

The partial exception is a pipeline that deploys the merged branch to an
ephemeral environment and runs checks against it. That is the right idea — this argument,
implemented in CI. The catch is what gets pointed at it. A preview environment
usually gets a smoke test written when the pipeline was built: the homepage
returns 200, maybe a health endpoint. It rarely gets the redirect chain, the
link graph, the cache headers or the API's response shape, because nobody went
back and added them. The environment is the easy half; the list of checks is the
half that decides whether it finds anything.

## Checking the artefact is indifferent to authorship

A request to `https://yoursite.example/plans` from a machine that has no
checkout, no session, no branch and no context returns a status code, a
document and a set of headers. That result is the same regardless of whether
one agent or four wrote the code behind it, whether they merged cleanly, and
whether anybody read the change.

That is the property worth having. The check does not need to know how the site
reached this state in order to report what it is now serving.

What that buys you, concretely — these are the checks we run, named as they
appear in HarpyWatch:

| The state you are worried about | What observes it |
| --- | --- |
| A URL you advertise that stopped answering after a merge | `http-uptime`, one check per address you nominate |
| Internal links to paths that moved | `broken-links` |
| A redirect one agent added and another inverted | `redirect-chain` |
| A page that assembles only after JavaScript runs, and now does not | `browser-render` |
| An API that still answers but no longer with the shape callers expect | `json-api`, `json-schema` |
| A header set on a branch preview and never on production | `security-headers`, `cache-policy` |

None of that is a substitute for review. It is a different question. Review
asks whether the change was a good idea; the check asks whether the site
currently works. Only the second has an answer a machine can produce on demand and repeatedly.
That machine fails in its own ways — its network, its own expired credentials,
its list of URLs going stale — but it fails visibly, and it does not get bored.

## Check the state before it is the one serving traffic

The cheapest moment to discover the merged-state 404 is before that state is
what customers get. HarpyWatch has no CI integration and no pre-deploy hook — it
points at URLs, and knows nothing about your branches. But a staging host is a
URL. If the merged branch is deployed somewhere before DNS
moves, the same set of checks runs against that address, and the difference
between the staging grid and the production grid is a description of what this
release changed about the observable site. That is a smaller, more honest thing
than a release note, and it is derived rather than written.

## The limit, stated

An external check cannot tell you that the merge was architecturally wrong, that
the new checkout step is confusing, or that the copy an agent rewrote now says
something the legal team would object to. It observes surfaces, from outside,
without context. Someone still has to read the parts of a change that require
judgement.

The argument is narrower: reading time is limited, and a dead link in a footer
is a poor use of it. Defects that a single unauthenticated request would reveal
are the ones to take off a person's plate first, and under concurrent agent authorship there are more of them, because there are
more merged states nobody looked at.

So keep the review, the tests, the branch protection and the staged rollout.
Just stop treating any of them as an answer to whether the site works right now,
because none of them was ever looking at the site. Something has to make the
request, from outside, on a schedule — and know as little about how the code got
there as a visitor does.