Watchdog

By the time we were running a dozen products across several servers, "is everything up?" had become a question with no single answer. Hosted monitoring services charge per check and per host, and none of them knew what our scheduler was supposed to be doing at 2 a.m.

We built the monitoring we wanted, and now every product on this site is covered by it.

• Catalog as code: every project has a file listing its services, processes, scheduled jobs, log files, certificates and URLs. A change to the system is a change to the file, reviewed like any other.
• Per-server agent that checks processes, disk, log freshness and rotation from inside the box.
• Heartbeats for scheduled jobs: a job that did not run is an incident, not a silence.
• HTTP, certificate and DNS checks from outside.
• Incidents with history, and email alerts that say what is wrong and where to look.
• Security checks: open ports, failed logins, package updates.

A central service holds the catalog and runs external checks; a small agent on each server reports in. The rule across the studio is that a new service, job or log file is not finished until Watchdog covers it, and that rule is enforced in review.

Watchdog covers every product on this site: the Axis web, worker and scheduler processes, BirdReel, Zoomin, the consumer apps, the nonprofit sites, their logs, their certificates and the 14,000 scheduled job runs a day. When a client chooses Build and run, their product is added to the same catalog.

Watchdog runs on its own server so that it is not down when the thing it monitors is. It is checked by an external uptime service, because somebody has to watch the watcher.