Why monitoring is an operating problem
For a SaaS company the infrastructure is the product. A slow API, a failed login, or a queue that stops draining is a customer incident, not an internal inconvenience. SOC 2, HIPAA, and ISO 27001 all expect you to show that you watch the systems that matter and that someone acts when they degrade. The frameworks do not name a vendor. They ask whether monitoring is defined, owned, and evidenced.
Buy tools after you can answer three questions. Which customer-facing paths have to work? What does "not working" look like in a metric, a log line, or a synthetic check? Who is allowed to be woken up, and what are they supposed to do first? A dashboard that nobody is accountable for is not a monitoring program.
Set the targets before the thresholds
Track uptime and availability against a target of 99.9%+. Track response time as p50, p95, and p99, not as an average that hides the slow tail. Track error rates, resource utilization (CPU, memory, disk, network), and the customer-impacting events that those host metrics are only a proxy for. Set thresholds that trigger investigation before customers notice degradation. A threshold copied from a vendor default is a guess. A threshold tied to the point where a real workflow gets worse is a control.
Page on customer harm. Open a ticket for conditions that are unhealthy but have not yet touched that harm. CPU at a high watermark on a batch worker is a ticket unless that worker sits on a request the customer is waiting for. An elevated error rate on the API those customers call is a page. If you cannot say which of the two an alert is, the alert is not ready to ship.
Monitoring checklist
Use this as a review of the program you have, not as a shopping list. A line you cannot answer with a named owner and a place to look is a gap.
- Infrastructure. Every production host, container service, database, queue, and load balancer emits CPU, memory, disk, and network, plus the saturation signal that actually fills up first: connection pool, queue depth, file descriptors, or inode use. You can name which metric means this tier is out of room and which metric is only context. Capacity is reviewed on a cadence, not discovered during an incident. A replica that has stopped replicating, a backup job that has stopped succeeding, and a certificate that will expire are monitored the same way as CPU. The primary looking healthy does not make those someone else's problem.
- Application performance monitoring. A slow or failed request can be traced across the services it touched, with time broken out so you can see whether it went to the application, the database, or a third party. p50, p95, and p99 are visible per endpoint customers actually call, not only as one site-wide number. Deploys are marked on the same graphs, so a regression has a suspect release. If APM covers one service and not the others on the path, say so. Do not describe the uncovered services as observed.
- Logging. Production logs land in one place you can query, with a retention period written down and defensible both to an investigator and to an auditor. A line that matters carries a request or trace id, a service name, and an environment. You can answer what a given request did without opening a shell on a host. Secrets, full payment numbers, and health data the request did not need are not logged because a debug statement was easy. Log volume has an owner. An unbounded debug level left on in production is a cost and a noise incident, not a setting.
- Synthetics and real user monitoring. Synthetic checks run from outside your network against the paths you would be embarrassed to hear about from a customer: login, the primary workflow, and a health check that fails when its dependencies fail. A static page that returns success while the API is down does not count. Checks run often enough that a broken deploy is caught before the next customer hits it. Real user monitoring covers what server metrics cannot see: a slow third-party script, a regional DNS failure, a browser-only error. When RUM and synthetics disagree, believe the signal closer to the user and then explain the gap. Do not page on a single probe from one region. Require a repeated failure or a second vantage point so one flaky probe network does not train the rotation to ignore pages.
- Alerting and on-call. Every alert has a severity, an owner, and a link to a runbook. Pages go to a rotation staffed for the hours you actually committed to. That commitment is business hours plus an on-call rotation sized to the engagement. It is not a claim that a person is watching a queue all night unless that is the rotation you hired and wrote down. The response target is the one in the agreement for that service, and you compare it with what happened. If one failure sends five pages, route the symptoms into one incident. Someone reviews what fired, on a schedule, and deletes or retunes the alerts that did not change a decision.
- Runbooks. Each paging alert points at a runbook a new on-call engineer can execute: what the alert means, how to confirm customer impact, the first checks, how to mitigate or roll back, and who to escalate to. Runbooks live next to the system they describe and change when that system changes. A wiki page last touched a year ago, and not linked from the alert, will not be read under pressure. A runbook that has never been used in a drill or a real incident is still a draft. After an incident, either update the runbook or record why you left it, in the ticket.
- Compliance evidence. You can show an auditor the list of what you monitor, who is accountable, a sample of alerts and the tickets they opened, the incident record for anything customer-facing, the runbook that was current, and the review where a threshold changed. SOC 2, HIPAA, and ISO 27001 evidence is that record across the audit window, not a screenshot taken the week the auditor is due. Access to the monitoring tools is limited and logged the same way as access to production. Retention and clock sync match the story in the control description. If the control says you review alerts monthly, the calendar invite and the notes exist for the months in the period.
Build it in an order that pays for itself
Start on the path that moves money or data
Instrument the request path customers pay for, the identity path, and the datastore under them. Host metrics on those tiers come first because they are cheap and they catch the outage where the process is simply gone. Leave internal tools until the customer path is boring to operate. An admin panel can wait.
Add application tracing where you cannot explain a slow request
If a customer says the product is slow and the host graphs are calm, you need a trace. Add application performance monitoring on the services in that path. Tag by endpoint and release. Do not tag by raw user id: the tool becomes a bill and a privacy problem. Mark deploys. A latency jump that lines up with a release is a rollback candidate before it is a research project.
Make logs answer the incident question
Agree a minimum shape for a log line and reject the rest in review. During an incident the question is almost always which requests failed, for whom, starting when. If that takes more than a query, logging is not done. Set retention to cover the investigation you expect and the audit window you committed to, then test that an older line is actually still there. A retention setting nobody has restored from is a hope.
Put a synthetic check on anything you would hate to learn from a ticket
Cover login, the equivalent of first run, and one transaction that proves the core loop. Run them from outside the network. A health endpoint that never touches the database will stay green through the outage you care about. Either make the check touch the dependency or do not let it be the only check. Add real user monitoring when you have browser clients. Server-side success with a client-side failure is a class of incident host metrics will not open for you.
Review the program on a calendar
Each month, look at what paged, what was noise, which runbooks were wrong, and whether the 99.9%+ target and the p50, p95, and p99 figures still describe the product you ship. Each quarter, walk the checklist and assign an owner to every gap. A monitoring program that is only edited during incidents accumulates alerts nobody believes.
Alerting decisions
Write the paging rule down. Page when a customer-facing path is failing or is about to, when data is at risk, or when the only copy of something important is unhealthy. Open a ticket when the system is degraded and customers are not, when capacity will run out on a planning horizon rather than tonight, or when a check failed once and has not repeated. Do not keep a page you have not acted on the last few times it fired. Delete it or fix the response.
Name severities in language the whole company shares. A sev-1 means customers cannot do the thing they came to do, or data integrity is in doubt, and the response target agreed for that service applies. Lower severities use a slower path. If they use the same notification as a sev-1, people stop trusting the page. Escalation is a list of humans and a way to reach them that you have tested, not a distribution list nobody reads.
Separate "the monitoring system is up" from "the product is up." If the collector dies, you want one alert that says you are blind, not silence. A synthetic check that does not depend on the same collector is the usual way to catch that blindness.
Common mistakes
- Paging on hosts instead of on the service. CPU, disk, and memory pages fire all week. The outage that matters is a customer path that stopped, and that alert was never created.
- Averages instead of percentiles. The median can look fine while p99 is a timeout. Keep p50, p95, and p99. Do not replace them with a mean and call the tail handled.
- Health checks that cannot fail. A static success response on a load balancer is not monitoring. The check has to fail when the dependency the user needs is down.
- Logs as a landfill. Everything is logged, nothing is queryable, and the bill shows up before the incident does. Log the decision points. Drop or sample the rest on purpose, and write down that choice.
- Runbooks detached from alerts. The wiki has a page. The alert does not link to it. Under pressure nobody goes looking.
- Evidence assembled the week of the audit. Exporting dashboards into a slide is not continuous monitoring. The tickets, the alert history, and the review notes are the evidence, and they have to exist through the period under review.
- Tool sprawl with no owner. Three products each cover a slice, nobody can see a request end to end, and each product pages a different channel. Pick the place an incident starts and route the rest there.
When the program needs another pair of hands
Keep the program inside the team if someone will still be editing alerts after the project excitement ends. Ask for help when the graphs exist but the pages are noise, when an audit is asking for monitoring evidence you cannot produce from tickets, or when the rotation is burning out on checks that do not map to customer impact. The work is the same either way: targets, the checklist above, runbooks, and a record. Some teams want the checks and the escalation path built and maintained while their own rotation keeps first response. Others want a partner to take the escalation, or to take first response, once the signal is worth waking someone for. That choice belongs in the statement of work, not in a tool default. The shape of that engagement is described under monitoring and incident response.
Our team works alongside yours to build monitoring programs that scale with your business. Book a Free Assessment to evaluate your current monitoring posture.