Your monitor cannot live in the thing it monitors

Some outages announce themselves. The worst ones arrive as an absence: the graphs stop, the checks stop reporting, and for the first several minutes the on-call engineer's only evidence that anything is wrong is a customer email.

When that happens, there is usually one reason. The thing doing the watching was standing on the thing that fell over.

A monitor is only as independent as its worst common dependency

Consider four common arrangements, in increasing order of sophistication. They fail in different ways, and they share one property.

A /healthz endpoint in the application. It reports the application's view of itself. When the process is dead, nothing serves the endpoint, and the answer is not "unhealthy" — there is no answer. Something else has to notice the absence, and that something else is usually the next item on this list.

A cron on the same box that curls the site and emails on failure. One host, two processes. Kernel panic, full disk, an OOM killer that took both processes, a systemctl stop in the wrong terminal — the site and its watcher die in the same instant.

An uptime pod in the same Kubernetes cluster. That survives a single node failing, which is real progress. It does not survive the ingress controller misrouting after a bad config reload, CoreDNS failing so that no pod can resolve anything, an expired kubelet client certificate taking nodes NotReady in a block, or the API server being unreachable so nothing gets rescheduled. Those are the failures that take out the whole service, and they take the watcher with it.

A monitoring stack in the same cloud region, on separate instances. Stronger still — and still sharing a region, an account, a network path and, usually, the same nameservers.

Each of these measures something real. The limitation is the question they are all asking: is the service healthy from inside the environment the service runs in. Outages have an inconvenient habit of being facts about that environment.

The domains that get held in common

It is worth naming them individually, because most setups eliminate one or two and quietly keep the rest.

The host. Covered above, and still the most common arrangement there is.

The network. A security group change, a firewall rule, a route table, a misapplied network policy. Internal traffic between your monitor and your app often takes a completely different path from a visitor's traffic — private subnet, service mesh, cluster DNS. The check can succeed over the path that still works while the path the public uses is severed.

DNS. This is the one that catches careful teams. Your internal check resolves api.internal or hits a service IP directly, or has the record in a local cache with a long TTL. A visitor resolves the public name through public resolvers. When a zone update goes wrong, when a registrar's nameserver delegation lapses, or when a DNSSEC signature expires, the internal check continues to pass with total confidence. Nothing is wrong with the server. The name simply no longer points at it.

The certificate. A check that connects over plain HTTP inside the cluster, or with verification disabled because the internal name does not match the certificate, cannot see an expired or misissued certificate on the public edge. TLS is terminated at a load balancer or a CDN that the internal check never touches. The socket a browser connects to is a different socket.

The cloud region. When a region's load balancer control plane, its managed database, or its object storage degrades, it takes the workload and everything you deployed alongside it — including the Prometheus that was going to tell you about it. This is the case people plan for on paper and rarely test, because testing it means already having something outside the region.

The account. The one that rarely makes it onto the diagram. A billing failure, a suspended account, an expired card, a compromised credential that gets everything disabled, an IAM policy change applied a bit broadly. When the account goes, every service in it goes, and that includes the alerting.

What an outside check sees that an inside one cannot

An external check is a request from a machine that has nothing in common with yours: a different network, a different resolver, no credentials, no privileged path, no knowledge of your topology. That poverty is the feature.

It resolves your public hostname the way a visitor does, so a broken delegation or an expired DNSSEC signature is a failed check rather than an invisible one. It completes a real TLS handshake against the public endpoint, so ssl-cert-expiry and cert-chain-trust read the certificate that was actually presented — not the file on disk, and not the internal certificate. It follows the redirect chain a browser follows, through the CDN and the load balancer and whatever else sits in front, so a loop introduced between two correctly behaving components shows up as a loop. It carries no cookie and no session, so a page that only works for someone already logged in fails, which is usually what you wanted to know.

And when your infrastructure is entirely gone, it keeps running, because it was never part of it. That is the point worth stating plainly: in the minutes when every tool inside your environment has stopped reporting, a check outside it is still in a position to answer. It has its own ways of failing — a provider outage, a routing change, an expired credential — and those are mostly not your ways of failing. Mostly, not entirely: if the check runs in the same region as the site, a regional event still takes both. Independence is a thing you get more or less of, and the amount you get is the whole benefit.

Be honest about what it gives up

External observation buys independence by giving up detail, and pretending otherwise would be dishonest.

An outside check sees a 500, not the exception that caused it. It cannot tell you which of four backends is failing, that the connection pool is exhausted, that a queue is backing up, or that a disk will be full in six hours. It has no per-request tracing, no logs, no memory profile. Its sample rate is a schedule, not every request, so a failure affecting one visitor in fifty may not land in its window — availability-sample exists to narrow that gap by probing repeatedly, and narrowing is not eliminating.

It is also blind to everything behind an authenticated boundary. Whatever your logged-in users see, an unauthenticated check does not.

So this is not an argument against internal instrumentation. Metrics, logs and traces are how you find out why, and nothing outside your network can replace them. The argument is narrower and firmer than that: the signal that tells you whether, and the alert that has to fire when everything is broken, cannot be hosted on the thing it is watching.

Two different tools, two different jobs. HarpyWatch is the outside one: the checks run from our infrastructure against your public hostnames, on a schedule, and they keep running when yours does not. If your alerting today would have gone quiet in any of the six failure domains above, that is worth fixing first, whoever you fix it with.