Infrastructure/Scheduling/ — Time-Dependent Operations & Retry Orchestration

Heartbeats Graceful Restart Backoff & Retry Delivery Recovery

Contents

Architecture

The scheduling sub-module is split into three pillars: heartbeat management (periodic keepalive signals), restart coordination (graceful process lifecycle), and delivery recovery (retry with backoff). Each pillar operates independently but shares common backoff and retry primitives.

flowchart TD
  subgraph Heartbeats["Heartbeat Management"]
    HR["heartbeat-runner.ts\nareHeartbeatsEnabled()\nsetHeartbeatsEnabled()"]
    HAH["heartbeat-active-hours.ts\nTime-based gating"]
    HRE["heartbeat-reason.ts\nReason classification"]
    HVI["heartbeat-visibility.ts\nChannel visibility rules"]
    HWK["heartbeat-wake.ts\nWake handlers"]
  end
  subgraph Restarts["Restart Coordination"]
    RST["restart.ts\nscheduleGatewaySigusr1Restart()\ndeferGatewayRestartUntilIdle()"]
    RESP["process-respawn.ts\nrestartGatewayProcessWithFreshPid()"]
    SUPV["supervisor-markers.ts\ndetectRespawnSupervisor()"]
  end
  subgraph Recovery["Delivery Recovery"]
    DQR["delivery-queue-recovery.ts\nrecoverPendingDeliveries()"]
    BOFF["backoff.ts\ncomputeBackoff()"]
    RTRY["retry.ts\nretryAsync()"]
  end
  HR --> HAH
  HR --> HRE
  HR --> HVI
  HR --> HWK
  RST --> RESP
  RESP --> SUPV
  DQR --> BOFF
  DQR --> RTRY
  RST --> BOFF

Data Flow

This sequence diagram shows how the three scheduling pillars interact during a typical gateway lifecycle: heartbeats run on their interval, a restart is requested and deferred until idle, and any failed deliveries trigger recovery with exponential backoff.

sequenceDiagram
    participant Config as Configuration
    participant HB as heartbeat-runner.ts
    participant Channel as Target Channel
    participant RST as restart.ts
    participant RESP as process-respawn.ts
    participant SUPV as supervisor-markers.ts
    participant DQR as delivery-queue-recovery.ts
    participant BOFF as backoff.ts

    Config->>HB: resolveHeartbeatIntervalMs(agentId, config)
    HB->>HB: areHeartbeatsEnabled() check
    HB->>HB: isHeartbeatEnabledForAgent(agentId)
    HB->>Channel: Send heartbeat pulse
    Channel-->>HB: HeartbeatEventPayload { status: "sent" }
    Note over HB,Channel: Repeats on resolved interval

    RST->>RST: setPreRestartDeferralCheck(fn)
    RST->>RST: deferGatewayRestartUntilIdle(opts)
    loop Deferral polling (max 300s)
        RST->>RST: Check in-flight messages
    end
    RST->>RST: authorizeGatewaySigusr1Restart(delayMs)
    RST->>RESP: restartGatewayProcessWithFreshPid()
    RESP->>SUPV: detectRespawnSupervisor(env, platform)
    SUPV-->>RESP: RespawnSupervisor | null
    RESP-->>RST: GatewayRespawnResult { mode, pid }

    DQR->>DQR: recoverPendingDeliveries(opts)
    loop Each failed delivery (max 5 retries)
        DQR->>DQR: isEntryEligibleForRecoveryRetry()
        DQR->>DQR: isPermanentDeliveryError() check
        DQR->>BOFF: computeBackoffMs()
        BOFF-->>DQR: delay [5s, 25s, 120s, 600s]
        DQR->>Channel: Retry delivery
    end
    DQR-->>RST: RecoverySummary { recovered, failed }

Heartbeat Pipeline

The heartbeat system maintains channel liveness by periodically emitting keepalive signals. The pipeline resolves whether heartbeats are globally enabled, whether a specific agent should participate, calculates the interval, determines visibility rules, and finally emits a HeartbeatEventPayload.

heartbeat-runner.ts

The core heartbeat orchestrator at 250+ lines. Exports the HeartbeatDeps dependency type which composes OutboundSendDeps, ChannelHeartbeatDeps, and optional runtime hooks (runtime?, getQueueSize?, nowMs?). Key functions:

Supporting Modules

Restart Coordination

Graceful restart is critical for zero-downtime deployments. The restart subsystem ensures in-flight messages are drained before the process is replaced, supports three OS-specific restart methods, and enforces cooldown periods to prevent restart storms.

restart.ts

The restart controller at 500+ lines. Manages the full lifecycle from deferral to signal emission:

Constants

process-respawn.ts

restartGatewayProcessWithFreshPid() returns a GatewayRespawnResult with one of four modes:

supervisor-markers.ts

detectRespawnSupervisor(env, platform) inspects environment variables and the OS platform to detect if the process is managed by launchd (macOS), systemd (Linux), or schtasks (Windows). Returns RespawnSupervisor | null.

Delivery Recovery

When message delivery fails, the recovery subsystem retries with exponential backoff, filters out permanent errors, and produces a summary of outcomes.

delivery-queue-recovery.ts

Core recovery orchestrator with MAX_RETRIES=5 and a backoff schedule of [5s, 25s, 120s, 600s]. Includes 14 compiled regex patterns for permanent error detection.

backoff.ts & retry.ts

Shared primitives used across all scheduling pillars:

Key Classes & Functions

Data Structures

Interactions

Configuration

Directory Map

scheduling/
├── heartbeat-runner.ts               ← Core heartbeat orchestrator (250+ lines)
├── heartbeat-active-hours.ts         ← Time-based gating for heartbeat windows
├── heartbeat-reason.ts               ← HeartbeatReasonKind classification
├── heartbeat-visibility.ts           ← Channel visibility rules
├── heartbeat-wake.ts                 ← Manual trigger / wake handlers
├── restart.ts                        ← Graceful restart controller (500+ lines)
├── process-respawn.ts                ← OS-aware process replacement
├── supervisor-markers.ts             ← launchd / systemd / schtasks detection
├── delivery-queue-recovery.ts        ← Failed delivery retry with backoff
├── backoff.ts                        ← BackoffPolicy & computeBackoff()
├── retry.ts                          ← Generic retryAsync<T>() wrapper
├── cron_expressions/                 ← Cron parsing & scheduling rules
│   └── ...
├── task_lifecycle/                   ← Scheduled task state machine
│   └── ...
└── job_queue/                        ← Priority job queue & workers
    └── ...

Deep-Dive Highlights

Core scheduling mechanisms have been extracted into dedicated focus pages for in-depth exploration. Each page covers one aspect of the scheduling pipeline with diagrams, code walks, and configuration details.

Deep-Dive Pages

Code Examples

Example: Resolving Heartbeat Summary for an Agent

import {
  resolveHeartbeatSummaryForAgent,
  areHeartbeatsEnabled,
  setHeartbeatsEnabled,
  type HeartbeatSummary,
} from "./heartbeat-runner";

// Ensure heartbeats are globally enabled
if (!areHeartbeatsEnabled()) {
  setHeartbeatsEnabled(true);
}

// Resolve full summary for a specific agent
const summary: HeartbeatSummary = resolveHeartbeatSummaryForAgent("agent-42");
console.log(summary);
// {
//   intervalMs: 60000,
//   reason: "cron-event",
//   visibility: "silent",
//   enabled: true,
// }

Example: Scheduling a Graceful Restart with Deferral

import {
  setPreRestartDeferralCheck,
  deferGatewayRestartUntilIdle,
  scheduleGatewaySigusr1Restart,
  type ScheduledRestart,
} from "./restart";

// Register a deferral check that blocks restart while messages are in flight
setPreRestartDeferralCheck(() => {
  const queueSize = getQueueSize();
  return queueSize === 0; // only allow restart when idle
});

// Defer until idle, then restart via SIGUSR1
const result: ScheduledRestart = await deferGatewayRestartUntilIdle({
  maxWaitMs: 120_000,  // wait up to 2 minutes
  reason: "deployment",
});

console.log(result);
// {
//   ok: true,
//   pid: 48291,
//   signal: "SIGUSR1",
//   delayMs: 0,
//   reason: "deployment",
//   mode: "signal",
//   coalesced: false,
//   cooldownMsApplied: 30000,
// }

Example: Recovering Failed Deliveries with Backoff

import {
  recoverPendingDeliveries,
  isPermanentDeliveryError,
  computeBackoffMs,
  type RecoverySummary,
} from "./delivery-queue-recovery";
import { computeBackoff, type BackoffPolicy } from "./backoff";
import { retryAsync, type RetryConfig } from "./retry";

// Check if an error is permanent (won't benefit from retry)
const permanent = isPermanentDeliveryError(new Error("channel_not_found"));
// true — matches one of the 14 permanent error regex patterns

// Compute backoff for a specific attempt
const policy: BackoffPolicy = {
  initialMs: 5000,
  maxMs: 600_000,
  factor: 5,
  jitter: true,
};
const delayMs = computeBackoff(policy, 3); // ~625s + jitter

// Recover all pending deliveries
const summary: RecoverySummary = await recoverPendingDeliveries({
  maxRetries: 5,
  dryRun: false,
});
console.log(summary);
// {
//   recovered: 12,
//   failed: 2,
//   skippedMaxRetries: 1,
//   deferredBackoff: 3,
// }

// Generic retry wrapper for arbitrary async operations
const data = await retryAsync(
  () => fetchWithTimeout("/api/status", 5000),
  { attempts: 3, minDelayMs: 300, maxDelayMs: 30_000, jitter: true },
);