Codegen Pipeline Focus Deep-Dive

🎯 The Problem

Four SDKs in four languages need to stay in lockstep with a JSON-RPC schema that evolves monthly. Humans cannot reliably keep 4 sets of type declarations in sync. Common failure modes without codegen: Node adds a parameter, Python is missed; Go emits a number where the schema demands an integer; .NET misses a nullable and crashes when the field is null. Each of these is a silent wire-level bug.

The codegen pipeline removes the human-memory dependency. Any schema change produces a diff in all four SDKs; if the diff is unexpected, the PR author fixes the schema (or the emitter). If the diff is missing, CI fails.

🔧 The Approach

A three-stage pipeline:

1. Normalise the schema so per-language emitters see a clean AST (inline small $refs, tag integer vs number, canonicalise nullable, patch "required-but-nullable" fields).
2. Walk the normalised tree once per language, writing idiomatic code via string concatenation + per-language name/quote helpers.
3. Format the output with each language's canonical formatter so diffs are deterministic.

Corrections are applied before normalisation so they look like schema edits rather than emitter overrides — this keeps emitters simple and the correction footprint obvious.

🔄 Why Not Alternatives?

Pipeline Diagram

flowchart TD
  S[api.schema.json
from @github/copilot/schemas]
  C[corrections/*.ts,.json
patches]
  P1[inlineSmallRefs]
  P2[tagIntegers]
  P3[canonicaliseNullable]
  P4[fixNullableRequiredRefs]
  AST[Shared AST
methods / types / events]
  FMT[Formatter
prettier / black / gofmt / dotnet-format]

  subgraph emit["Emitters"]
    E1[typescript.ts]
    E2[python.ts]
    E3[go.ts]
    E4[csharp.ts]
  end

  S --> P1
  C --> P1
  P1 --> P2 --> P3 --> P4 --> AST
  AST --> E1 --> FMT --> O1[nodejs/src/generated/rpc.ts]
  AST --> E2 --> FMT --> O2[python/copilot/_generated_rpc.py]
  AST --> E3 --> FMT --> O3[go/rpc/generated_rpc.go]
  AST --> E4 --> FMT --> O4[dotnet/src/Generated/Rpc.cs]

Step-by-Step

// scripts/codegen/index.ts
const schemaPath = process.env.SCHEMA_PATH ??
    path.join(__dirname, "../../nodejs/node_modules/@github/copilot/schemas/api.schema.json");
const raw = JSON.parse(fs.readFileSync(schemaPath, "utf8"));

const corrected = applyCorrections(raw, loadCorrections());
// corrections can: override a type, add description, delete a method, pin a field to integer

const normalised = fixNullableRequiredRefsInApiSchema(postProcessSchema(corrected));

const ast = buildAst(normalised);
// ast.methods[n] = { name, params, result, description }
// ast.types[n]   = { name, fields, kind: "record" | "union" | "enum" }
// ast.events[n]  = { tag, fields }

await emitTypeScript(ast, { out: "nodejs/src/generated/rpc.ts" });
await emitPython(ast, { out: "python/copilot/_generated_rpc.py" });
await emitGo(ast, { out: "go/rpc/generated_rpc.go", skipEvents: true });
await emitCSharp(ast, { out: "dotnet/src/Generated/Rpc.cs" });
await emitCSharp(ast, { out: "dotnet/src/Generated/SessionEvents.cs", eventsOnly: true });

await runFormatter("prettier", "nodejs/src/generated/rpc.ts");
await runFormatter("black",    "python/copilot/_generated_rpc.py");
await runFormatter("gofmt",    "go/rpc/generated_rpc.go");
await runFormatter("dotnet",   "dotnet/src/Generated/*.cs");

npm run codegen -- --check
# Generates into .tmp/, diffs against committed files, exits 1 on any difference.

Type Mapping Cheatsheet

* Go could emit time.Duration but currently uses int64 ms; future emitter work. ** C# uses MillisecondsTimeSpanConverter to bridge wire ms to TimeSpan.

⚖️ Complexity & Tradeoffs

The whole pipeline runs in ~1.5s for all four languages. Not optimised — emitters use string += patterns and run sequentially. Could be parallelised but there's no pressure.

The expensive cost isn't time but intellectual: each emitter must understand its target language's idioms deeply. Adding a fifth language (e.g., Rust, Java) would mean writing a fifth emitter of ~500-800 lines. The shared infrastructure (schema loading, AST, corrections) is reused, so it's not starting from zero — but it's not free either.

💥 Failure Recovery

Codegen is not a runtime path — no user ever runs it. Failures manifest as: (1) local npm run codegen exits non-zero; (2) CI --check step fails the PR; (3) compile failures in whichever SDK consumed a malformed output. The repo's scripts/check-api-surface.ts is a secondary guard — it compares the Node public API to a committed manifest, catching accidentally-exported renames.

Edge Cases

Configuration

Directory Map

scripts/codegen/
├── index.ts               ← orchestrator entry
├── utils.ts               ← name helpers, AST walkers
├── postProcess.ts         ← postProcessSchema + fixNullableRequiredRefs
├── typescript.ts          ← TS emitter
├── python.ts              ← Py emitter
├── go.ts                  ← Go emitter
├── csharp.ts              ← C# emitter
└── corrections/           ← patches

Related

• JSON-RPC Transport — the wire format these RPCs ride on.
• Protocol Negotiation — where SDK_PROTOCOL_VERSION ends up.