---
title: "Your monitor cannot live in the thing it monitors"
description: "A health check inside your own stack goes silent at exactly the moment it had something to say. Here are the failure domains an internal check shares with the site, and what an external one sees instead."
url: https://harpywatch.com/blog/your-monitor-cannot-live-in-the-thing-it-monitors
kind: article
author: "The HarpyWatch team"
published: 2026-09-02T18:06:09.575253+00:00
tags: ["Monitoring", "Reliability"]
publisher: "HarpyWatch"
---

# Your monitor cannot live in the thing it monitors

Some outages announce themselves. The worst ones arrive as an absence: the
graphs stop, the checks stop reporting, and for the first several minutes the
on-call engineer's only evidence that anything is wrong is a customer email.

When that happens, there is usually one reason. The thing doing the watching
was standing on the thing that fell over.

## A monitor is only as independent as its worst common dependency

Consider four common arrangements, in increasing order of sophistication. They
fail in different ways, and they share one property.

A `/healthz` endpoint in the application. It reports the application's view of
itself. When the process is dead, nothing serves the endpoint, and the answer
is not "unhealthy" — there is no answer. Something else has to notice the
absence, and that something else is usually the next item on this list.

A cron on the same box that curls the site and emails on failure. One host, two
processes. Kernel panic, full disk, an OOM killer that took both processes, a
`systemctl stop` in the wrong terminal — the site and its watcher die in the
same instant.

An uptime pod in the same Kubernetes cluster. That survives a single node
failing, which is real progress. It does not survive the ingress controller
misrouting after a bad config reload, CoreDNS failing so that no pod can
resolve anything, an expired kubelet client certificate taking nodes `NotReady`
in a block, or the API server being unreachable so nothing gets rescheduled.
Those are the failures that take out the whole service, and they take the
watcher with it.

A monitoring stack in the same cloud region, on separate instances. Stronger
still — and still sharing a region, an account, a network path and, usually,
the same nameservers.

Each of these measures something real. The limitation is the question they are
all asking: *is the service healthy from inside the environment the service
runs in*. Outages have an inconvenient habit of being facts about that
environment.

## The domains that get held in common

It is worth naming them individually, because most setups eliminate one or two
and quietly keep the rest.

**The host.** Covered above, and still the most common arrangement there is.

**The network.** A security group change, a firewall rule, a route table, a
misapplied network policy. Internal traffic between your monitor and your app
often takes a completely different path from a visitor's traffic — private
subnet, service mesh, cluster DNS. The check can succeed over the path that
still works while the path the public uses is severed.

**DNS.** This is the one that catches careful teams. Your internal check
resolves `api.internal` or hits a service IP directly, or has the record in a
local cache with a long TTL. A visitor resolves the public name through public
resolvers. When a zone update goes wrong, when a registrar's nameserver
delegation lapses, or when a DNSSEC signature expires, the internal check
continues to pass with total confidence. Nothing is wrong with the server. The
name simply no longer points at it.

**The certificate.** A check that connects over plain HTTP inside the cluster,
or with verification disabled because the internal name does not match the
certificate, cannot see an expired or misissued certificate on the public
edge. TLS is terminated at a load balancer or a CDN that the internal check
never touches. The socket a browser connects to is a different socket.

**The cloud region.** When a region's load balancer control plane, its managed
database, or its object storage degrades, it takes the workload and everything
you deployed alongside it — including the Prometheus that was going to tell you
about it. This is the case people plan for on paper and rarely test, because
testing it means already having something outside the region.

**The account.** The one that rarely makes it onto the diagram. A billing failure, a
suspended account, an expired card, a compromised credential that gets
everything disabled, an IAM policy change applied a bit broadly. When the
account goes, every service in it goes, and that includes the alerting.

## What an outside check sees that an inside one cannot

An external check is a request from a machine that has nothing in common with
yours: a different network, a different resolver, no credentials, no privileged
path, no knowledge of your topology. That poverty is the feature.

It resolves your public hostname the way a visitor does, so a broken delegation
or an expired DNSSEC signature is a failed check rather than an invisible one.
It completes a real TLS handshake against the public endpoint, so
`ssl-cert-expiry` and `cert-chain-trust` read the certificate that was actually
presented — not the file on disk, and not the internal certificate. It follows
the redirect chain a browser follows, through the CDN and the load balancer and
whatever else sits in front, so a loop introduced between two correctly
behaving components shows up as a loop. It carries no cookie and no session, so
a page that only works for someone already logged in fails, which is usually
what you wanted to know.

And when your infrastructure is entirely gone, it keeps running, because it was
never part of it. That is the point worth stating plainly: **in the minutes
when every tool inside your environment has stopped reporting, a check outside
it is still in a position to answer.** It has its own ways of failing — a
provider outage, a routing change, an expired credential — and those are mostly
not your ways of failing. Mostly, not entirely: if the check runs in the same
region as the site, a regional event still takes both. Independence is a thing
you get more or less of, and the amount you get is the whole benefit.

## Be honest about what it gives up

External observation buys independence by giving up detail, and pretending
otherwise would be dishonest.

An outside check sees a 500, not the exception that caused it. It cannot tell
you which of four backends is failing, that the connection pool is exhausted,
that a queue is backing up, or that a disk will be full in six hours. It has no
per-request tracing, no logs, no memory profile. Its sample rate is a schedule,
not every request, so a failure affecting one visitor in fifty may not land in
its window — `availability-sample` exists to narrow that gap by probing
repeatedly, and narrowing is not eliminating.

It is also blind to everything behind an authenticated boundary. Whatever your
logged-in users see, an unauthenticated check does not.

So this is not an argument against internal instrumentation. Metrics, logs and
traces are how you find out *why*, and nothing outside your network can replace
them. The argument is narrower and firmer than that: the signal that tells you
**whether**, and the alert that has to fire when everything is broken, cannot
be hosted on the thing it is watching.

Two different tools, two different jobs. HarpyWatch is the outside one: the
checks run from our infrastructure against your public hostnames, on a
schedule, and they keep running when yours does not. If your alerting today
would have gone quiet in any of the six failure domains above, that is worth
fixing first, whoever you fix it with.