Agents/Providers — Model Selection

Model Resolution Fallback Chains Context Windows Error Recovery

Contents

Overview

flowchart TD
  subgraph Config["Configuration Sources"]
    AGENT["agents.list[].model\nAgent config"]
    HOOK["resolveHookModelSelection()\nPlugin hooks"]
    OVERRIDE["Per-run model override"]
    DEFAULT["Default model\n(system fallback)"]
  end
  subgraph Resolution["Model Resolution Pipeline"]
    CHECK["Check per-run override"]
    PLUGIN["Check plugin hooks"]
    RESOLVE["resolveEffectiveRuntimeModel()"]
    CONTEXT["Calculate ContextWindowInfo"]
  end
  subgraph Fallback["Fallback Chain"]
    FB["model-fallback.ts"]
    CLASSIFY["Error classification"]
    NEXT["Next model in chain"]
    COMPACT["Context compaction"]
    PROBE["Cooldown probe"]
  end
  subgraph Output["Result"]
    MODEL["effectiveModel +\nContextWindowInfo"]
    ERROR["FallbackSummaryError\nattempts[] + soonestCooldownExpiry"]
  end
  OVERRIDE --> CHECK
  CHECK -->|"present"| RESOLVE
  CHECK -->|"absent"| PLUGIN
  PLUGIN -->|"hook returns model"| RESOLVE
  PLUGIN -->|"no hook"| AGENT
  AGENT --> RESOLVE
  DEFAULT -.->|"last resort"| RESOLVE
  RESOLVE --> CONTEXT
  CONTEXT --> MODEL
  MODEL -->|"API error"| FB
  FB --> CLASSIFY
  CLASSIFY -->|"rate_limit"| NEXT
  CLASSIFY -->|"context_overflow"| COMPACT
  CLASSIFY -->|"timeout"| PROBE
  CLASSIFY -->|"auth"| NEXT
  NEXT --> RESOLVE
  COMPACT --> RESOLVE
  PROBE --> RESOLVE
  NEXT -->|"chain exhausted"| ERROR

The resolution pipeline checks configuration sources in priority order: per-run override first, then plugin hooks via resolveHookModelSelection(), then the agent’s configured model, and finally the system default. Once resolved, resolveEffectiveRuntimeModel() calculates the effective context window. If the model fails at runtime, the fallback chain in model-fallback.ts classifies the error and selects an appropriate recovery strategy.

Decision Points

How It Works

Model selection involves three distinct phases: resolution (determining the initial model), context sizing (calculating effective window), and fallback (recovering from runtime failures).

Phase 1: Model Resolution

The resolveHookModelSelection() function in run/setup.ts checks agent configuration and plugin hooks in priority order to determine the starting model for a run.

// run/setup.ts — model resolution pipeline
export async function resolveHookModelSelection(
  agentConfig: AgentConfig,
  runParams: RunParams,
  pluginHooks: PluginHooks,
): Promise<ResolvedModel> {
  // Priority 1: Per-run model override
  if (runParams.modelOverride) {
    return resolveModelFromRegistry(runParams.modelOverride);
  }

  // Priority 2: Plugin hook (can inspect task context)
  if (pluginHooks.resolveModel) {
    const hookResult = await pluginHooks.resolveModel({
      agent: agentConfig,
      task: runParams.task,
      history: runParams.history,
    });
    if (hookResult?.model) {
      return resolveModelFromRegistry(hookResult.model);
    }
  }

  // Priority 3: Agent config model
  if (agentConfig.model) {
    return resolveModelFromRegistry(agentConfig.model);
  }

  // Priority 4: System default
  return resolveModelFromRegistry(DEFAULT_MODEL);
}

Phase 2: Context Window Sizing

Once the model is resolved, resolveEffectiveRuntimeModel() calculates the effective context window. The result is a ContextWindowInfo object used by the streaming pipeline to manage token budgets.

// run/setup.ts — context window calculation
export function resolveEffectiveRuntimeModel(
  resolved: ResolvedModel,
  thinkingLevel: ThinkingLevel,
): EffectiveRuntimeModel {
  const baseWindow = resolved.contextWindow;

  // Thinking modes reserve a portion of the context window
  const thinkingReserve = getThinkingReserve(thinkingLevel, baseWindow);

  // Effective window = base - thinking reserve - safety margin
  const effectiveWindow = Math.floor(
    (baseWindow - thinkingReserve) / SAFETY_MARGIN,
  );

  return {
    effectiveModel: resolved.modelId,
    provider: resolved.provider,
    contextWindow: {
      total: baseWindow,
      effective: effectiveWindow,
      thinkingReserve,
      safetyMargin: SAFETY_MARGIN,
    },
  };
}

interface ContextWindowInfo {
  total: number;        // Raw model context window (e.g., 200000)
  effective: number;    // Usable tokens after reserves
  thinkingReserve: number; // Tokens reserved for thinking/reasoning
  safetyMargin: number;    // Buffer multiplier (1.2 = 20%)
}

Phase 3: Fallback Chains

The model-fallback.ts module implements cascading fallback when a model fails at runtime. Error classification drives the fallback strategy:

Fallback Cascade State Diagram

stateDiagram-v2
  [*] --> PrimaryModel : Agent run starts
  PrimaryModel --> Success : API success
  PrimaryModel --> ErrorClassify : API error

  ErrorClassify --> TryNextModel : rate_limit / auth
  ErrorClassify --> TryCompaction : context_overflow
  ErrorClassify --> RetryWithProbe : timeout

  TryNextModel --> FallbackModel1 : Chain position 2
  FallbackModel1 --> Success : API success
  FallbackModel1 --> ErrorClassify2 : API error

  ErrorClassify2 --> FallbackModel2 : Chain position 3
  FallbackModel2 --> Success : API success
  FallbackModel2 --> ChainExhausted : No more fallbacks

  TryCompaction --> CompactedRetry : History compacted
  CompactedRetry --> Success : API success
  CompactedRetry --> TryNextModel : Still overflows

  RetryWithProbe --> ProbedRetry : Short timeout retry
  ProbedRetry --> Success : API success
  ProbedRetry --> TryNextModel : Probe failed

  ChainExhausted --> FallbackSummaryError : All options exhausted

  Success --> [*]
  FallbackSummaryError --> [*]

  note right of FallbackSummaryError
    Contains:
    - attempts: FallbackAttempt[]
    - soonestCooldownExpiry: number | null
  end note

The state diagram shows the full fallback cascade. Each failed attempt is recorded as a FallbackAttempt in the summary. If all options are exhausted, a FallbackSummaryError is thrown containing the full attempt history and the soonest cooldown expiry time across all tried providers.

Annotated Code Walkthrough

Walk through the fallback chain logic in model-fallback.ts.

// model-fallback.ts — fallback orchestrator
export async function runWithFallback<T>(
  chain: FallbackChain,
  execute: (model: ResolvedModel) => Promise<T>,
): Promise<T> {
  const attempts: FallbackAttempt[] = [];
  let soonestCooldown: number | null = null;

  for (const entry of chain.models) {
    try {
      const resolved = resolveModelFromRegistry(entry.modelId);
      const result = await execute(resolved);
      return result; // Success — return immediately
    } catch (err) {
      const category = classifyError(err);
      attempts.push({
        modelId: entry.modelId,
        error: category,
        timestamp: Date.now(),
      });

      // Track cooldown for retry-after info
      if (category === "rate_limit" && err.retryAfter) {
        const expiry = Date.now() + err.retryAfter * 1000;
        if (!soonestCooldown || expiry < soonestCooldown) {
          soonestCooldown = expiry;
        }
      }

      // Context overflow: try compaction before next model
      if (category === "context_overflow" && !entry.compactionAttempted) {
        entry.compactionAttempted = true;
        // Re-queue same model after compaction
        chain.models.splice(chain.models.indexOf(entry) + 1, 0, entry);
        continue;
      }

      // Auth errors: skip entire provider
      if (category === "auth") {
        markAuthProfileFailure(entry.modelId, err);
        continue;
      }
    }
  }

  throw new FallbackSummaryError(attempts, soonestCooldown);
}

// model-fallback.ts — error classification
function classifyError(err: unknown): ErrorCategory {
  if (err instanceof ProviderError) {
    switch (err.status) {
      case 429: return "rate_limit";
      case 401:
      case 403: return "auth";
      case 413: return "context_overflow";
      default:
        if (err.status >= 500) return "server_error";
    }

    // Check error message for context overflow signals
    if (err.message?.includes("context_length_exceeded")) {
      return "context_overflow";
    }
    if (err.message?.includes("rate limit")) {
      return "rate_limit";
    }
  }

  if (err instanceof TimeoutError) return "timeout";

  return "unknown";
}

// model-fallback.ts — cooldown probe
export function shouldAllowCooldownProbeForReason(
  reason: ErrorCategory,
  lastFailure: FallbackAttempt,
): boolean {
  // Only allow probes for transient errors
  if (reason !== "rate_limit" && reason !== "timeout") {
    return false;
  }

  // Must have been at least 10s since last failure
  const elapsed = Date.now() - lastFailure.timestamp;
  if (elapsed < 10_000) return false;

  // Only one probe per cooldown cycle
  if (lastFailure.probeAttempted) return false;

  lastFailure.probeAttempted = true;
  return true;
}

OpenRouter Capability Resolution

When the resolved model is served through OpenRouter, openrouter-model-capabilities.ts fetches capability flags from OpenRouter’s API to determine which features the underlying model supports:

// openrouter-model-capabilities.ts — capability resolution
interface OpenRouterCapabilities {
  supportsToolCalls: boolean;
  supportsStreaming: boolean;
  supportsJsonMode: boolean;
  maxContextWindow: number;
  maxOutputTokens: number;
}

export async function resolveOpenRouterCapabilities(
  modelId: string,
): Promise<OpenRouterCapabilities> {
  const metadata = await fetchOpenRouterModelMetadata(modelId);
  return {
    supportsToolCalls: metadata.supported_parameters?.includes("tools") ?? false,
    supportsStreaming: metadata.supported_parameters?.includes("stream") ?? true,
    supportsJsonMode: metadata.supported_parameters?.includes("response_format") ?? false,
    maxContextWindow: metadata.context_length ?? 4096,
    maxOutputTokens: metadata.top_provider?.max_completion_tokens ?? 4096,
  };
}

Forward Compatibility Testing

The model.forward-compat.test.ts file ensures that new models added to the registry don’t break existing resolution logic. These tests verify that:

• Unknown model IDs fall through to the default model gracefully
• Provider normalization handles new vendor prefixes without crashing
• Fallback chains work correctly when new models are inserted at any position
• Context window calculations remain accurate for new model sizes

Edge Cases & Error Handling

Configuration

Related Topics

See also:

• Providers overview — the parent sub-module page
• Adapter Pattern — how resolved models map to stream wrappers
• Token Management — auth cooldowns that interact with fallback decisions
• Streaming pipeline — where context window info drives compaction

Directory Map

pi-embedded-runner/
├── model.ts                              ← Model registry & resolution (24 KB)
├── model.provider-normalization.ts       ← Provider ID normalization
├── model-fallback.ts                     ← Cascading fallback strategy
├── model.forward-compat.test.ts          ← Forward compatibility tests
├── openrouter-model-capabilities.ts      ← OpenRouter capability flags
└── run/
    └── setup.ts                          ← resolveHookModelSelection(), resolveEffectiveRuntimeModel()