Health Checks

Overview

Health checks serve two primary purposes: providing a readiness signal for load balancers and orchestrators (Kubernetes, systemd), and offering internal diagnostics for operators monitoring the Gateway's subsystems. The health system runs as a CronService task that periodically evaluates all registered health probes and caches the aggregated result.

External consumers (HTTP health endpoints, monitoring dashboards) call getHealthCache() to retrieve the latest cached HealthSummary without triggering a fresh evaluation. The getHealthVersion() function returns a monotonically increasing counter that increments each time the health status changes, allowing efficient polling via ETags or long-polling.

Health Check Cycle

The following flowchart illustrates the complete health check cycle, from scheduled trigger through cache update and version increment.

flowchart TD
    A[CronService Trigger] --> B[Collect Health Probes]
    B --> C{All probes registered?}
    C -- No --> D[Log warning: missing probes]
    D --> E[Run available probes]
    C -- Yes --> E
    E --> F[Evaluate each probe]
    F --> G{Any probe failed?}
    G -- Yes --> H[Aggregate: status = unhealthy]
    G -- No --> I{Any probe degraded?}
    I -- Yes --> J[Aggregate: status = degraded]
    I -- No --> K[Aggregate: status = healthy]
    H --> L[Build HealthSummary]
    J --> L
    K --> L
    L --> M[Compare with cached summary]
    M --> N{Status changed?}
    N -- Yes --> O[Increment healthVersion]
    O --> P[Update healthCache]
    N -- No --> P
    P --> Q[Log health result]
    Q --> R[Schedule next check]

HealthSummary Type

The HealthSummary is the primary data structure returned by getHealthCache(). It contains the overall status, individual probe results, timestamps, and diagnostic metadata used by monitoring systems.

type HealthStatus = "healthy" | "degraded" | "unhealthy";

interface HealthSummary {
  status: HealthStatus;
  version: number;
  checkedAtMs: number;
  durationMs: number;
  probes: HealthProbeResult[];
  metadata: {
    gatewayId: string;
    uptime: number;
    activeConnections: number;
    pendingRequests: number;
  };
}

interface HealthProbeResult {
  name: string;
  status: HealthStatus;
  message: string;
  durationMs: number;
  lastSuccessMs: number | null;
  consecutiveFailures: number;
}

// The cache accessor — returns the latest cached summary or null if
// no health check has run yet (e.g., during initial boot)
function getHealthCache(): HealthSummary | null {
  return healthCacheStore.get();
}

// Returns a monotonically increasing version counter that increments
// each time the health status changes (not on every check cycle)
function getHealthVersion(): number {
  return healthCacheStore.getVersion();
}

How It Works

interface HealthProbe {
  name: string;
  check: () => Promise<HealthProbeResult>;
  timeoutMs: number;
  critical: boolean;
}

class HealthCheckManager {
  private probes: Map<string, HealthProbe> = new Map();
  private cache: HealthSummary | null = null;
  private version: number = 0;

  registerProbe(probe: HealthProbe): void {
    if (this.probes.has(probe.name)) {
      logHealth.error(`Duplicate probe registration: ${probe.name}`);
      return;
    }
    this.probes.set(probe.name, probe);
  }

  async runAllProbes(ctx: GatewayRequestContext): Promise<HealthSummary> {
    const start = Date.now();
    const results: HealthProbeResult[] = [];

    for (const [name, probe] of this.probes) {
      try {
        const result = await Promise.race([
          probe.check(),
          timeout(probe.timeoutMs, name),
        ]);
        results.push(result);
      } catch (err) {
        results.push({
          name,
          status: "unhealthy",
          message: `Probe threw: ${String(err)}`,
          durationMs: Date.now() - start,
          lastSuccessMs: null,
          consecutiveFailures: 1,
        });
      }
    }

    const overallStatus = this.aggregateStatus(results);
    const summary: HealthSummary = {
      status: overallStatus,
      version: this.version,
      checkedAtMs: Date.now(),
      durationMs: Date.now() - start,
      probes: results,
      metadata: {
        gatewayId: ctx.gatewayId,
        uptime: Date.now() - ctx.startedAtMs,
        activeConnections: ctx.nodeRegistry.size(),
        pendingRequests: ctx.metrics.getPendingCount(),
      },
    };

    // Only increment version on status change
    if (!this.cache || this.cache.status !== summary.status) {
      this.version++;
      summary.version = this.version;
    }

    this.cache = summary;
    return summary;
  }

  private aggregateStatus(results: HealthProbeResult[]): HealthStatus {
    if (results.some(r => r.status === "unhealthy")) return "unhealthy";
    if (results.some(r => r.status === "degraded")) return "degraded";
    return "healthy";
  }
}

// Registered during initializeServices() in boot sequence
cronService.registerTask(
  "health-check",
  "*/30 * * * * *",
  async () => {
    const summary = await healthCheckManager.runAllProbes(ctx);

    if (summary.status === "unhealthy") {
      logHealth.error(
        `Health check FAILED: ${summary.probes
          .filter(p => p.status === "unhealthy")
          .map(p => p.name)
          .join(", ")}`
      );
    }

    ctx.logger.debug(
      `Health check: ${summary.status} (v${summary.version}, ${summary.durationMs}ms)`
    );
  }
);

httpRouter.get("/health", (req, res) => {
  const cached = getHealthCache();
  if (!cached) {
    res.status(503).json({ status: "not-ready", message: "Health check pending" });
    return;
  }

  const statusCode = cached.status === "healthy" ? 200
    : cached.status === "degraded" ? 200
    : 503;

  res.setHeader("ETag", `"v${cached.version}"`);
  res.setHeader("Cache-Control", "no-cache, must-revalidate");
  res.status(statusCode).json(cached);
});

httpRouter.get("/health/version", (req, res) => {
  res.json({ version: getHealthVersion() });
});

const logHealth: { error: (message: string) => void } = {
  error: (message: string) => {
    const entry = {
      level: "error",
      category: "health",
      timestamp: new Date().toISOString(),
      message,
    };
    process.stderr.write(JSON.stringify(entry) + "\n");

    // Also increment the health error counter for metrics
    ctx.metrics.increment("health.errors");
  },
};

Health Check Sequence

This sequence diagram shows a complete health check cycle including probe execution, cache update, and external consumer access.

sequenceDiagram
    participant CS as CronService
    participant HM as HealthCheckManager
    participant WS as WebSocket Probe
    participant DB as Session Store Probe
    participant NR as Node Registry Probe
    participant HC as Health Cache
    participant LB as Load Balancer

    CS->>HM: runAllProbes(ctx)

    par Probe execution
        HM->>WS: check()
        WS-->>HM: {status: "healthy", durationMs: 2}
    and
        HM->>DB: check()
        DB-->>HM: {status: "healthy", durationMs: 15}
    and
        HM->>NR: check()
        NR-->>HM: {status: "degraded", message: "3 stale nodes"}
    end

    HM->>HM: aggregateStatus() → "degraded"
    HM->>HM: Compare with previous status
    alt status changed
        HM->>HM: Increment version
    end
    HM->>HC: Update cache with new HealthSummary

    Note over LB: Periodic readiness probe
    LB->>HC: GET /health
    HC-->>LB: 200 {status: "degraded", version: 42}
    LB->>LB: Mark Gateway as degraded

Built-in Health Probes

The Gateway ships with several built-in health probes that are automatically registered during boot. Custom probes can be added via the registerProbe() API.

Cache Invalidation Strategy

The health cache uses a write-on-schedule pattern — it is only updated when the CronService triggers a new check cycle. Between cycles, the cache returns the same HealthSummary instance. The version field allows efficient change detection.

Integration with GatewayRequestContext

The health system is wired into the GatewayRequestContext during boot. The context holds references to the HealthCheckManager, the cache accessor functions, and the logHealth logger. This integration ensures that all subsystems can contribute health probes and query health status without importing global singletons.

// Fields added to GatewayRequestContext for health integration
interface GatewayRequestContext {
  // ... other fields ...

  healthCache: {
    get: () => HealthSummary | null;
    getVersion: () => number;
    invalidate: () => void;
  };

  healthCheckManager: HealthCheckManager;

  logHealth: {
    error: (message: string) => void;
  };
}

// During buildGatewayRequestContext():
const healthCheckManager = new HealthCheckManager();
const ctx: GatewayRequestContext = {
  // ...
  healthCache: {
    get: () => healthCheckManager.getCache(),
    getVersion: () => healthCheckManager.getVersion(),
    invalidate: () => healthCheckManager.clearCache(),
  },
  healthCheckManager,
  logHealth,
};

Error Handling & Edge Cases

Configuration Reference

Related Topics

• Boot Overview — full boot subsystem documentation
• Startup Sequence — how health probes are registered during boot
• Capability Negotiation — node-registry probe monitors connected nodes

Directory Map

src/gateway/
├── health.ts                  ← HealthCheckManager, getHealthCache(), getHealthVersion()
├── health-probes.ts           ← Built-in probe implementations
├── health-types.ts            ← HealthSummary, HealthProbeResult, HealthStatus
├── health-log.ts              ← logHealth interface and error stream
├── cron.ts                    ← CronService, health check task scheduling
└── context.ts                 ← GatewayRequestContext health integration