Focus

Task Lifecycle

Process supervision, graceful restarts, and cross-platform daemon management

Table of Contents

Overview & Motivation

Process supervision ensures the OpenClaw gateway stays running across crashes, config changes, and planned restarts. The lifecycle handles graceful shutdown (waiting for in-flight work), cross-platform daemon management (launchd, systemd, schtasks), and audit logging for every restart event.

💡 Analogy — Hospital Shift Change

Think of restarts like a hospital's shift change protocol. The outgoing doctor (current process) can't leave until all critical patients are stable (deferral polling). The incoming doctor (new process) gets authorized credentials (SIGUSR1 auth token) with a 5-second window. A supervisor (launchd/systemd) ensures someone is always on duty. The shift change is logged for the record (audit trail).

Restart Controller

500+ line orchestrator handling SIGUSR1 signaling, cooldown, and deferral-until-idle

Process Respawn

Four respawn modes — spawned, supervised, disabled, failed — with PID tracking

Supervisor Detection

Auto-detects launchd, systemd, or schtasks via environment variable patterns

Gateway Lock

PID + timestamp file lock with /proc liveness checks, 30s stale TTL, SHA256 config hash

Restart Controller — restart.ts

The primary restart controller at 500+ lines is the most complex module in the scheduling subsystem. It coordinates the entire restart lifecycle: authorization gates, deferral polling, signal emission, cooldown management, and audit logging.

Core Types

Three key interfaces define the restart domain model:

Authorization Flow

Not every component may trigger a restart. The authorization system uses a token-based approach with a 5-second grace period:

⚠️ Security Note

The 5-second authorization window is intentionally short. In production, a race between authorize() and consume() wider than 5 s indicates a hung process and triggers an alert via the monitoring pipeline.

Deferral & Cooldown

deferGatewayRestartUntilIdle(opts) implements a polling loop that waits for all pending work to drain before allowing the restart to proceed. The loop polls every 500 ms with exponential backoff up to a 5-minute ceiling. If work never drains, the restart is forced after the ceiling.

scheduleGatewaySigusr1Restart(opts?) wraps the signal emission with a 30-second cooldown. Multiple restart requests arriving within the cooldown are coalesced — only one actual signal is sent, and the ScheduledRestart.coalesced count records how many were merged.

Code Walk — Restart Scheduling with Cooldown

1

Check if a restart is already pending within the cooldown window.

2

If coalescing, increment the counter and return early.

3

Otherwise, authorize the restart, wait for idle, then emit SIGUSR1.

4

Record the cooldown timestamp and audit info for the event.

interface ScheduledRestart {
  ok: boolean;
  pid: number;
  signal: 'SIGUSR1';
  delayMs: number;
  reason?: string;
  mode: 'emit' | 'signal';
  coalesced: number;
  cooldownMsApplied: number;
}

const COOLDOWN_MS = 30_000;
let lastRestartTs = 0;
let coalescedCount = 0;

async function scheduleGatewaySigusr1Restart(
  opts?: { reason?: string; audit?: RestartAuditInfo }
): Promise<ScheduledRestart> {
  const now = Date.now();
  if (now - lastRestartTs < COOLDOWN_MS) {
    coalescedCount++;
    return {
      ok: true, pid: process.pid, signal: 'SIGUSR1',
      delayMs: 0, reason: opts?.reason,
      mode: 'emit', coalesced: coalescedCount,
      cooldownMsApplied: COOLDOWN_MS - (now - lastRestartTs),
    };
  }

  authorizeGatewaySigusr1Restart(5_000);
  await deferGatewayRestartUntilIdle({ maxWaitMs: 300_000 });

  emitGatewayRestart();  // sends SIGUSR1
  lastRestartTs = Date.now();
  const result: ScheduledRestart = {
    ok: true, pid: process.pid, signal: 'SIGUSR1',
    delayMs: Date.now() - now, reason: opts?.reason,
    mode: 'signal', coalesced: coalescedCount,
    cooldownMsApplied: COOLDOWN_MS,
  };
  coalescedCount = 0;
  return result;
}

Process Respawn — process-respawn.ts

restartGatewayProcessWithFreshPid() handles the actual OS-level process creation. It returns a GatewayRespawnResult with one of four modes:

Supervisor Detection

detectRespawnSupervisor(env, platform) inspects environment variables to determine which process supervisor, if any, is managing the gateway:

When a supervisor is detected, the respawn mode switches to "supervised" — the process simply exits with code 0, trusting the supervisor to restart it.

Gateway Lock — gateway-lock.ts

The gateway lock (260 lines) prevents multiple gateway instances from running simultaneously. The lock file contains the owning PID and a creation timestamp. On Linux, liveness is verified by reading /proc/${pid}/stat. On other platforms, a port-based check is used as fallback.

ℹ️ Lock Details

Lock parameters: 30s stale TTL, 100ms poll interval, 5s acquisition timeout. The lock path includes a SHA256 hash of the config path, allowing multiple gateway instances with different configs on the same machine.

interface GatewayLockHandle {
  lockPath: string;
  configPath: string;
  release: () => Promise<void>;
}

class GatewayLockError extends Error {
  constructor(
    message: string,
    public readonly lockPath: string,
    public readonly ownerPid?: number
  ) {
    super(message);
    this.name = 'GatewayLockError';
  }
}

async function acquireGatewayLock(opts?: {
  configPath: string;
  staleTtlMs?: number;
  pollIntervalMs?: number;
  timeoutMs?: number;
}): Promise<GatewayLockHandle> {
  const staleTtl = opts?.staleTtlMs ?? 30_000;
  const pollInterval = opts?.pollIntervalMs ?? 100;
  const timeout = opts?.timeoutMs ?? 5_000;
  const hash = createHash('sha256')
    .update(opts!.configPath)
    .digest('hex')
    .slice(0, 12);
  const lockPath = join(stateDir, `gateway-${hash}.lock`);
  const deadline = Date.now() + timeout;

  while (Date.now() < deadline) {
    const existing = await tryReadLock(lockPath);
    if (!existing || !isProcessAlive(existing.pid, staleTtl)) {
      await writeLock(lockPath, process.pid);
      return {
        lockPath,
        configPath: opts!.configPath,
        release: () => unlink(lockPath),
      };
    }
    await sleep(pollInterval);
  }
  throw new GatewayLockError('Lock acquisition timed out', lockPath);
}

Restart Lifecycle Diagram

The following state diagram captures the complete restart lifecycle, from the initial request through authorization, deferral polling, signal emission, respawn, and back to normal operation.

Click to expand 
stateDiagram-v2
    [*] --> Running
    Running --> RestartRequested: config change / API call
    RestartRequested --> AuthorizationCheck: validate caller
    AuthorizationCheck --> Rejected: token expired / unauthorized
    Rejected --> Running: log & resume
    AuthorizationCheck --> DeferralPolling: authorized
    DeferralPolling --> DeferralPolling: pending work exists (500ms backoff)
    DeferralPolling --> SIGUSR1Emit: idle or max wait exceeded
    SIGUSR1Emit --> ProcessExiting: signal sent
    ProcessExiting --> SupervisorRespawn: supervisor detected
    ProcessExiting --> SelfRespawn: no supervisor
    SupervisorRespawn --> Running: new PID via launchd/systemd
    SelfRespawn --> Running: detached child spawned
    ProcessExiting --> Terminated: OPENCLAW_NO_RESPAWN set

Platform-Specific Restart Paths

Each platform has a distinct restart strategy dictated by the available process supervisor. The flowchart below shows the decision tree used by triggerOpenClawRestart().

Click to expand 
flowchart TD
    A["triggerOpenClawRestart()"] --> B{"detectRespawnSupervisor()"}
    B -->|launchd| C["launchctl kickstart -k"]
    B -->|systemd| D["systemctl restart openclaw"]
    B -->|schtasks| E["schtasks /Run /TN task"]
    B -->|none| F{"OPENCLAW_NO_RESPAWN?"}
    F -->|set| G["Exit code 0 — no respawn"]
    F -->|unset| H["spawn detached child"]
    C --> I["RestartAttempt { ok, method: launchctl }"]
    D --> J["RestartAttempt { ok, method: systemd }"]
    E --> K["RestartAttempt { ok, method: schtasks }"]
    H --> L["GatewayRespawnResult { mode: spawned, pid }"]
    G --> M["GatewayRespawnResult { mode: disabled }"]

Audit Trail

Every restart event records a RestartAuditInfo object persisted to the state directory. This includes the actor (user or system), device ID, client IP (for remote-triggered restarts), and a list of changed config paths that prompted the restart. The audit log is append-only and rotated at 10 MB.

ℹ️ Debugging Tip

To inspect recent restarts, run openclaw gateway audit --last 10. This reads the audit log and pretty-prints the last 10 entries with timestamps, reasons, and whether the restart was coalesced.