environments — RL Training Environments

🎯 Why This Module Exists

Without the environments module, every RL training pipeline would need to implement its own agent loop, tool dispatch, reward verification, and sandbox isolation from scratch. The core Hermes agent (run_agent.py) is designed for interactive CLI use — single query, single session. RL training requires batch rollouts across hundreds of parallel tasks, each with isolated sandboxes, configurable tool distributions, and reward scoring. This module bridges that gap by wrapping the agent's tool-calling engine in Atropos-compatible BaseEnv subclasses. It provides ToolContext for reward verifiers to inspect rollout state (e.g., "did the model actually create that file?"), manages thread pool executors for concurrent tool calls, and abstracts away Phase 1 (OpenAI server) vs Phase 2 (VLLM ManagedServer) differences. The benchmark environments (TerminalBench, TBLite, YCBench, SWE, WebResearch) provide standardized evaluation harnesses, while the 11 tool call parsers handle the diverse output formats of models like Hermes, DeepSeek, Qwen, LLaMA, Mistral, GLM, and Kimi.

🚶 Step-by-Step Narrated Walkthrough

Follow a single RL training rollout for a TerminalBench task as it flows through the entire environments pipeline:

1. Rollout Request — Atropos framework calls get_next_item() on the TerminalBench environment, requesting a new task from the dataset.
2. Dataset Item — The environment returns a dataset item containing a terminal challenge prompt (e.g., "Find all Python files larger than 1MB in /data").
3. Prompt Formatting — format_prompt() converts the dataset item into an OpenAI-format user message with system prompt and enabled toolsets.
4. Toolset Sampling — HermesAgentEnvConfig samples the tool distribution for this rollout group. All rollouts in the same group share the same tool configuration for fair comparison.
5. Agent Loop Start — HermesAgentLoop begins the multi-turn loop, calling the LLM with the tools= parameter listing available tool schemas.
6. Tool Call Parsing — The appropriate parser (e.g., HermesParser for Hermes models) extracts structured tool calls from the LLM response. Phase 2 mode uses client-side parsing for lower latency.
7. Tool Execution — Each tool call is dispatched to handle_function_call() in a ThreadPoolExecutor (128 workers). The tool runs in a sandbox scoped to the rollout's task_id.
8. Loop Continuation — Tool results are appended to the conversation history. The loop continues until max_agent_turns is reached or the model stops calling tools.
9. AgentResult Capture — The full conversation history, turns used, tool errors, and reasoning per turn are captured atomically in an AgentResult dataclass.
10. ToolContext Creation — A ToolContext is created for the reward verifier, giving it unrestricted access to all Hermes tools scoped to this rollout's task_id.
11. Reward Computation — compute_reward() uses the ToolContext to verify the model's work (e.g., run the same command and check output), returning a score from 0.0 to 1.0.
12. Cleanup and Scoring — Sandbox resources (terminal session, browser profile) are cleaned up in a try-finally block. The ScoredDataGroup is built from ManagedServer state and returned to Atropos for training.

Tool Call Parsing Strategy

The Problem: Different LLMs produce tool calls in radically different formats — XML tags, JSON blocks, structured outputs, custom delimiters. The agent loop needs a unified interface.

The Approach: A parser registry maps model identifiers to parser classes. Each parser implements extract_tool_calls(text) -> List[ToolCall]. Phase 2 mode selects the parser at environment init based on model config; Phase 1 relies on the OpenAI API's native parsing.

Why This Algorithm: A single regex-based parser would break on edge cases across 11 formats. Dedicated parsers per model family isolate format-specific logic. The registry pattern keeps the agent loop agnostic to model details.

graph TD
    A["Raw LLM Output"] --> B{"Phase 1 or 2?"}
    B -->|"Phase 1"| C["OpenAI API Native Parsing"]
    B -->|"Phase 2"| D["Parser Registry Lookup"]
    D --> E["Model-Specific Parser"]
    E --> F["extract_tool_calls()"]
    C --> G["Normalized ToolCall List"]
    F --> G
    G --> H["Agent Loop Dispatch"]

Hot Paths

The most performance-sensitive code paths in this module:

• Tool dispatch via ThreadPoolExecutor: O(1) submission, but tool execution time varies from milliseconds (file read) to minutes (terminal commands). The 128-worker pool is sized for high-concurrency benchmarks.
• Tool call parsing: O(n) per response where n is response length. Dedicated parsers avoid regex backtracking on complex outputs.
• Reward verification: Variable cost — may involve running test suites, compiling code, or browsing. Scoped by ToolContext to prevent accidental global state access.

Resource Usage

• Memory: Each rollout holds its AgentResult (full conversation history). For 30-turn rollouts with tool outputs, this can be 1-5 MB per rollout. Freed after scoring.
• I/O: Heavy terminal and file I/O during tool execution. Sandboxed per task_id. Network I/O for LLM calls (Phase 1) or shared memory for VLLM (Phase 2).
• Threads: 128-thread pool shared across all environments. resize_tool_pool() allows per-environment tuning.