test — Conformance Suite

🎯 Why This Module Exists

Four SDKs + one protocol = a matrix of 4 implementations × N scenarios of "do we match the CLI?". Writing 4 copies of "send prompt, check response" tests would mean tests drift. A single set of YAML scenarios consumed by four harnesses enforces that the behaviour contract is one artefact — and any discrepancy surfaces as a normalised diff in CI, not as "oh, Go doesn't emit that event." This is also the mechanism used during protocol-version bumps: add the new scenario once; all four SDKs must implement and pass, or the PR is blocked.

🏗️ Design Decisions

Harness Architecture

flowchart LR
  YAML[test/snapshots/<feature>/*.yaml
scenario + expected events]
  CLI[copilot CLI
bundled binary]

  subgraph harnesses["Per-SDK harnesses"]
    NH[Node: test/harness.ts]
    PH[Python: test/harness.py]
    GH[Go: go/internal/e2e]
    CH[.NET: dotnet/test/Harness]
  end

  subgraph sdks["SDKs"]
    NODE[nodejs/ Client/Session]
    PY[python/ Client/Session]
    GO[go/ Client/Session]
    DOT[dotnet/ Client/Session]
  end

  YAML --> NH
  YAML --> PH
  YAML --> GH
  YAML --> CH

  NH --> NODE
  PH --> PY
  GH --> GO
  CH --> DOT

  NODE --> CLI
  PY --> CLI
  GO --> CLI
  DOT --> CLI

  CLI -->|events| NODE
  CLI -->|events| PY
  CLI -->|events| GO
  CLI -->|events| DOT

  NODE -->|normalise + diff| REPORT[test report]
  PY -->|normalise + diff| REPORT
  GO -->|normalise + diff| REPORT
  CH -->|normalise + diff| REPORT

🚶 One Harness Run

Key Components

⚡ Normalisation Algorithm

Each harness applies a fixed set of replacements to the event stream before diffing:

• sessionId → <SESSION_ID>
• messageId → <MESSAGE_ID_N> (indexed to preserve cross-references within the stream)
• traceId, spanId → redacted
• timestamp → omitted
• Numeric durations → omitted (timing varies)
• Tool call IDs → <TOOL_CALL_ID_N>

The normaliser is careful to preserve structural relationships (a toolCallCompleted referencing TOOL_CALL_ID_2 must refer to the same call as the earlier toolCallStarted). Violations of this invariant are themselves a test failure.

Adding a Scenario

1. Create test/snapshots/<feature>/my_case.yaml with scenario: + expectedEvents: sections.
2. Run Node harness in record mode: npm run test:record -- my_case. Captures and writes expected events.
3. Run all four harnesses in check mode: npm run test:conformance. All must pass.
4. Submit PR; reviewer sees the YAML change + any generated code updates.

⚠ What Could Go Wrong

📊 Performance

Each scenario spawns a CLI, runs one session, tears down. Typical wall clock: 0.5-3s per scenario per SDK. Running all four harnesses over 20 scenarios parallelised completes in under a minute. CI parallelises by SDK (not by scenario) because CLI reuse within a harness saves cold-start cost.

Deep-Dive Highlights

Configuration Reference

Code Examples

A scenario YAML (illustrative)

scenario:
  name: deny_then_approve
  prompt: "Run ls in /tmp"
  tools:
    - name: shell
      parameters: { type: object, properties: { cmd: { type: string } } }
  permissionResponses:
    - match: { tool: shell }
      response: deny
    - match: { tool: shell }
      response: approve
expectedEvents:
  - type: assistantMessage
    content: { $match: "..." }
  - type: permissionRequested
    tool: shell
  - type: permissionDenied
  - type: permissionRequested
    tool: shell
  - type: permissionApproved
  - type: toolCallStarted
    id: <TOOL_CALL_ID_1>
  - type: toolCallCompleted
    id: <TOOL_CALL_ID_1>
  - type: sessionIdle

Normaliser snippet

function normalise(events: SessionEvent[]): SessionEvent[] {
    const idMap = new Map<string, string>();
    return events.map(e => {
        const copy = structuredClone(e);
        if ("timestamp" in copy) delete copy.timestamp;
        if ("sessionId" in copy) copy.sessionId = "<SESSION_ID>";
        if ("messageId" in copy) copy.messageId = mapId(idMap, "MSG", copy.messageId);
        if ("toolCallId" in copy) copy.toolCallId = mapId(idMap, "TOOL_CALL", copy.toolCallId);
        return copy;
    });
}

Node harness core

export async function runScenario(scenario: Scenario): Promise<Result> {
    const client = new CopilotClient();
    await client.start();
    const session = await client.createSession({
        tools: scenario.tools,
        onPermissionRequest: mkPermissionHandler(scenario.permissionResponses),
    });
    const actual: SessionEvent[] = [];
    session.on(e => actual.push(e));
    await session.send({ prompt: scenario.prompt });
    await session.onIdle();
    await client.stop();
    return diff(normalise(actual), scenario.expectedEvents);
}

Directory Map

test/
├── snapshots/
│   ├── permissions/                   ← permission-flow scenarios
│   ├── hooks_extended/                ← pre/post-tool hooks
│   ├── session_fs/                    ← virtual FS provider
│   ├── commands/                      ← slash commands
│   ├── system_message/                ← replace/customize/transform
│   ├── basic/                         ← smoke-test scenarios
│   └── ...
├── harness.ts                         ← Node harness (reference)
├── harness.py                         ← Python harness
├── normaliser.ts                      ← Node normalisation helpers
├── README.md                          ← Author's guide
└── package.json

go/internal/e2e/                       ← Go harness (integrates with `go test`)
dotnet/test/Harness/                   ← .NET harness (xUnit)