---
title: The Health Check Hierarchy
description: "Liveness, readiness, deep health — what each health check level tells you and why you need all three for a travel booking platform that cannot afford downtime."
canonical: https://nowah.xyz/blog/health-check-hierarchy
lastModified: "2026-08-07T03:52:24.387Z"
---

# The Health Check Hierarchy

Liveness, readiness, deep health — what each health check level tells you and why you need all three for a travel booking platform that cannot afford downtime.

The container is running. The process started. The health check returns 200 OK. And the service can't reach the database. Every request that needs data fails with a connection error, but the health check keeps saying everything is fine.

This is what happens when your health check only verifies that the process is alive. It answers "is this thing running?" but not "can this thing do useful work?" The difference between those two questions is the difference between a healthy infrastructure and one that lies to you.

We run three levels of health checks, each answering a different question, each used by a different system.

## Liveness: is the process running?

![Illustration for this section](https://pics.nowah.xyz/website-media/infrastructure-028-img-1.webp)

The liveness check is at \`GET /health/live\`\. It does almost nothing\. It receives the request and returns 200 OK\. No database call\. No cache check\. No external service verification\. Just "yes, the HTTP server is listening and can respond\."

The container orchestrator uses the liveness probe to detect stuck processes. If a Node.js process deadlocks (a synchronous operation blocks the event loop, a memory leak causes thrashing, or an unhandled exception leaves the process in a broken state), the liveness probe times out. The orchestrator kills the container and starts a fresh one.

[Liveness probes](/blog/health-checks-liveness-probes-ai-services) should be fast and have no external dependencies. If your liveness check queries the database, and the database is down, the orchestrator kills your containers. Now you have no running containers and a database outage. When the database recovers, there's nothing running to serve traffic. The liveness check made the outage worse.

Keep liveness simple. Process alive? 200. Process dead? Timeout. That's it.

## Readiness: can it accept traffic?

The readiness check sits between liveness and deep health. It answers: is this service ready to handle requests? A service might be alive (process running) but not ready (still loading data, database migration in progress, warming up caches).

During startup, the service goes through an initialization phase. Database connections are established, cache connections are opened, reference data is loaded. During this phase, the liveness probe passes (the process is running) but the readiness probe fails (the service isn't ready to handle traffic yet).

The orchestrator uses the readiness probe to decide whether to route traffic to this container. A container that's alive but not ready gets no traffic. Once it reports ready, traffic starts flowing.

This prevents a common failure mode: a newly deployed container starts receiving traffic before its database connection pool is established. The first batch of requests all fail with connection errors, then the pool warms up and everything is fine. With readiness probes, the container only receives traffic once the pool is ready.

## Deep health: are all dependencies reachable?

![Supporting diagram](https://pics.nowah.xyz/website-media/infrastructure-028-img-2.webp)

Our deep health check is at \`GET /health\`\. It verifies every critical dependency:

**Database connectivity.** The check runs a simple query against the database. If the query succeeds, the database is reachable. If it fails or times out, the database dependency is degraded.

**Cache connectivity.** A ping to the cache layer verifies it's reachable and responsive.

**External service connectivity.** We verify that critical external services (like our [agentic memory](/blog/agentic-memory-smarter-over-time) service) are reachable. Not that they're returning correct data, just that the network path works and the service responds.

Each dependency is checked independently, and the response includes the status and latency of each check. A deep health response looks like:

```
{
 "status": "healthy",
 "database": { "status": "connected", "latency_ms": 3 },
 "cache": { "status": "connected", "latency_ms": 1 },
 "memory_service": { "status": "connected", "latency_ms": 45 }
}
```

If any dependency is unreachable, the overall status changes to "degraded" and the specific failing dependency is identified. The response target is under 200 milliseconds for the complete deep check.

The deep health check is used by monitoring systems, not by the container orchestrator. Monitoring systems poll this endpoint and alert when dependencies degrade. The orchestrator uses the simpler liveness and readiness probes for container lifecycle decisions.

## When health checks lie

The hardest failure mode is a health check that passes when it shouldn't. The database is reachable, but all queries are slow. The cache is connected, but its eviction policy changed and it's returning stale data. The external service responds to pings but returns errors on actual requests.

These are false-positive health checks. The system reports healthy, but it's not functioning correctly.

We mitigate this by making health checks slightly more meaningful than a connectivity ping. Our database health check runs an actual query (a simple SELECT), not just a connection check. If the query takes longer than a threshold, the check reports degraded even though the connection itself succeeded.

But there are limits to how much you can test in a health check. You can't run a full integration test on every health check poll. The check itself would be too slow and resource-intensive. The health check catches the gross failures (dependency unreachable, connectivity broken). Finer-grained quality issues (slow queries, stale cache data, degraded external service quality) are caught by application-level monitoring and alerting.

## Health check design patterns

If you're implementing health checks for your own services, here are the patterns that work.

**Separate endpoints for each check level.** Don't combine liveness and deep health on the same endpoint. The orchestrator needs to call liveness frequently (every few seconds) with low overhead. The monitoring system can call deep health less frequently (every 30 seconds) with more overhead.

**Set timeouts on health check responses.** A health check that hangs for 30 seconds is nearly as bad as one that fails. Set a strict timeout (we use 5 seconds for deep health, 1 second for liveness) and report failure if the timeout is exceeded.

**Include individual dependency status.** Don't just return a boolean. Return the status of each dependency. When the overall check reports degraded, the responder shouldn't have to guess which dependency is the problem.

**Don't let health checks cause load.** If your health check runs a database query, make it a lightweight one\. Don't run a complex aggregation\. Don't scan a large table\. A simple \`SELECT 1\` is enough to prove connectivity\.

**Test your health checks regularly.** Health checks that work when everything is fine might not work correctly when things are broken, which is exactly when you need them most. Periodically take a dependency down in staging and verify that the health check correctly reports the failure.

---

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).