Designing worlds for testing agents
How Ancha should serve event-driven agents completely, and leave room for autonomous ones.
In short
- The eval loop is well understood: tasks, repeated trials, graders on the end state. What teams can't get is a world to run it in, with realistic state, people and systems that react, and time.
- Today, everything that can happen in an Ancha world is hard-coded in the engine. That is the main problem.
- Fix it with one block: a reactive world kernel, where every reaction is a pluggable reactor that acts through the same APIs as the agent. People, CI, faults, webhooks, time compression, grading, generation and forking then fall out as reactors or views.
- Build level 2 (event-driven agents) completely on that kernel. Give level 3 (autonomous agents) interfaces, not features.
What teams actually do
Anthropic's eval guidance describes the loop teams converge on. Start with 20–50 tasks drawn from real failures. Run each task several times, because agents are nondeterministic. Grade the outcome, meaning the environment's end state, rather than the agent's claims or the exact path it took. Give every trial a clean environment, and read transcripts to tell agent errors from grader errors. The trials run in a harness the team already has or picks: Harbor, Braintrust, LangSmith, Langfuse, Arize, or plain pytest.
Agents reach SaaS tools through MCP servers or direct APIs. When the agent loop runs in a hosted harness such as Claude Managed Agents, private tools are reached through MCP tunnels or self-hosted sandboxes.
So running trials is not where teams get stuck. They get stuck on four things:
- building a realistic world for every task;
- the other side's behavior: customers, teammates, approvers, CI;
- fidelity, because a world that behaves unlike production makes the eval lie;
- turning a production failure into a task they can rerun.
The product is the world, not the runner.
Three levels of agent complexity
The same playground is used very differently depending on the agent.
| Level 1 · Textbook | Level 2 · Event-drivenWe build for this | Level 3 · AutonomousWe leave room for this | |
|---|---|---|---|
| Example | Customer support | On-call SRE, sales (SDR), pull-request agents | OpenClaw-style personal agents |
| Trigger | A user message; a few turns | Webhooks, alerts, schedules; minutes to days | Wakes itself on a heartbeat and finds its own work; weeks |
| Other side | A simulated customer with goals, hidden facts and moods | Teammates, approvers, prospects and CI, all reacting | The owner, everyone in their life, strangers, other agents |
| Twins | Shallow: orders, refunds, tickets | Deep, consistent across tools, with cause and effect | Wide: mail, calendar, chat, files, payments, browser, shell |
| Time | Barely matters | Compressed waits: CI takes 10 minutes, a reply comes in 2 days | A week in minutes; the agent's own timers follow the world clock |
| Adversity | Tricky or manipulative customers | Rate limits, outages, duplicate webhooks, missing scopes | Prompt injection in email, phishing, malicious skills, memory poisoning |
| Grading | End state, policy and tone; pass^k | Outcome, process rules (paged the right team, asked before deleting), time to resolve | Long-run outcomes, integrity, attack success rate, owner satisfaction |
| Trials | Hundreds of short conversations | Dozens of medium scenarios | A few long lives, with save points and branches |
At level 1 the hard part is the simulated customer, not the API. Level 2 is where twin depth and cross-tool cause and effect matter, and where the gap is widest: vendor test accounts are faithful but don't react or reset, and static mocks don't change when the agent acts. Level 3 needs a whole simulated life. Vending-Bench 2 shows why integrity must be graded too: when only the final balance counted, one model reportedly forged emails and withheld refunds.
The main problem
In Ancha today, what can happen in a world is fixed in the engine: actors that post at fixed times, a few hard-wired mechanisms (monitor → incident → Slack; merge → deploy), and a closed YAML vocabulary. Every new realistic reaction is engine work: a reviewer asking for changes, CI failing on a bad commit, a prospect replying two days later, a burst of 429s. Level 2 is made of exactly these reactions.
So the block to get right is a reactive world kernel. Do it well and the other blocks become plugins.
Agent under testruns in the team's own harness
calls through REST or MCP, receives webhooks
World
Twin APIsSlack · GitHub · Datadog · PagerDuty · …
reads and writes
Shared stateone store per world
every call and every change becomes an event
Event log and clockthe kernel
reactors subscribe
Reactorspeople · CI and deploys · faults · adversary · agent timersThey act through the same twin APIs as the agent.
out
Outputsend state · diff · timeline · grades
The kernel's contract
- One store per world holds every tool's state, with one clock and one append-only event log.
- Everything that happens is an event: the agent's calls, state changes, deliveries, the passage of time.
- A reactor subscribes to events and acts through the same twin APIs as the agent, under a named identity. It can also act as the platform, for things no user does through the API: CI finished, a payment disputed.
- The whole world serializes, including state, clock, pending events and reactor memory, so it can be forked at any event.
What falls out
| Block | On the kernel |
|---|---|
| Simulated people, at every level | Reactors with an identity: rules first, LLM personas with record-and-replay |
| CI, deploys, metrics, webhooks to the agent | Platform reactors |
| Faults: 429, 503, latency, incidents, expired tokens | Reactors on the request path and on state |
| Time compression | The clock drains the event queue; the agent's own timers become events |
| Grading and explanations | Queries over the event log plus an end-state diff, joined with the agent's transcript into one timeline |
| Scenario generation | A generated event history before t = 0, frozen by hash |
| Production import | Recorded traces translated into events and replayed |
| Red teaming | An adversary reactor that picks events to make the agent fail |
| Fork and replay | Serialize the world at any event |
| Any harness | The same verbs exposed as an API |
Engineering tradeoffs
| Decision | Options | Our call | Why |
|---|---|---|---|
| Who runs trials | Our runner, or the team's harness | A world-first API; our runner is a thin client | Trials already have homes; the world is what's missing |
| How behavior is defined | Closed YAML mechanisms, or reactors in code | Reactors in TypeScript and Python; YAML stays as shorthand for common rules | New reactions shouldn't need engine changes |
| Simulated people | Timelines, rules, or LLM personas | Rules first; LLM personas with record-and-replay | Realism where it matters, reproducible regressions |
| Determinism | Live generation, or frozen | Frozen by hash for regression runs, live for exploration; generated content is labeled | The agent is nondeterministic already; the world shouldn't add noise |
| Building twins | Hand-written, spec-generated, or an LLM long tail | A spec-driven core, hand-tuned semantics where agents actually call, conformance probes against real test accounts, and an LLM long tail that is frozen and labeled | Depth where it's used, and never a silent 200 |
| Reach | SDK base URL, MCP, network intercept, hosted | Base URL plus MCP with the official tool names for the level-2 stack; intercept inside containers; hosted later | Agents use both paths; hosted harnesses need MCP or a public endpoint |
| Process model | One store per world, or one process per tool | One store per world | Cross-tool consistency and cheap forks |
| Grading | Paths, or outcomes | End state and diff first; event-log rules for process constraints; rubric judges for text; report pass@k and pass^k | Grade outcomes, not paths |
| What speed buys | Wall time, or cost | Cost and parallelism | Model calls dominate a trial's wall time |
| Where it runs | Local, or hosted | Local for development and CI; hosted when customers' harnesses run in the cloud | Keep the seam; build when pulled |
Level 2, complete. Level 3, room to grow.
Build fully for level 2
- One level-2 stack, deep. For on-call, that means Datadog, PagerDuty, Slack, and GitHub including
gh,gitand GraphQL. - People: rule reactors and LLM personas (teammates, approvers, prospects), replayable.
- Platform behavior: CI, deploys, metrics, and webhooks to the agent's endpoint.
- Faults at three layers (transport, business, auth), plus approvals and scopes.
- A world API with create, seed, advance, inject, state, diff, fork, reset and events, over REST and MCP, with TypeScript and Python clients and a CLI.
- Grading: end-state predicates, diffs, event-log rules and rubric judges; pass@k and pass^k; adapters for pytest, vitest and Harbor.
- Scenarios generated from a description and frozen, or imported from a real incident.
Leave interfaces for level 3
- Agent clock: the agent's heartbeat and timers subscribe to the world clock.
- New kinds of twin (filesystem, shell, browser, a simulated web) behind the same twin contract.
- Long-lived personas with memory, and an owner simulator that sets goals and judges outcomes.
- Adversary reactors with an injection library.
- Hosted reach through MCP tunnels and public endpoints.
- Consequence metrics, such as money, satisfaction and time to resolve, kept by reactors.
Each of these is a new reactor or a new kind of twin. None should need a kernel change; that is the test of the kernel design.
Where the code stands
Keep
- One store per world, with the clock and IDs inside the snapshot
- A clean world for every trial
- End-state checks
- Uncovered calls fail loudly (501) and count as coverage gaps
- Sealed container mode, and agents as webhook endpoints
Not doing
- Silent stubs
- Per-minute metering inside CI
- LLM-generated worlds that aren't frozen
- Chasing twin count
Change
- The product center is a YAML scene with its own runner
- Actors post at fixed times and never react; no LLM-driven people
- Behavior comes from a few hard-wired mechanisms
- 11 hand-written twins, of which only 4 work in scenes
- REST on loopback only; MCP only for Slack, with our own tool names
- Small hand-written seeds
- Evidence from the tool side only; the agent's transcript isn't joined
- Not yet validated against a live LLM agent
Open questions
- Which level-2 stack comes first: on-call, sales, or code review?
- How many target agents reach tools through MCP rather than direct APIs?
- Do level-2 customers need hosted reach on day one?
- How do these teams test today, and exactly where do they stop? Watch a few before rebuilding.
Reading
Anthropic
- Demystifying evals for AI agents: tasks, trials, graders, transcripts and outcomes; pass@k vs pass^k; grade outcomes, not paths.
- Effective harnesses for long-running agents: initializer and coding agents; end-to-end testing tools changed the results.
- Writing effective tools for agents, with agents: eval-driven tool work in realistic, complex environments.
- Building agents with the Claude Agent SDK: gather context, act, verify; MCP for Slack and Asana.
- Building effective agents: workflows versus agents.
- Claude Managed Agents: self-hosted sandboxes and MCP tunnels.
Benchmarks that already simulate the other side
- τ²-bench: a model plays the user, and both sides act on shared state.
- TheAgentCompany: a simulated software company with colleagues played by an LLM.
- AppWorld: everyday apps with hundreds of APIs, simulated users' digital lives, and state-based tests.
- Vending-Bench 2: a year-long simulated business, graded on the final balance.
Level 3 context
- OpenClaw glossary: heartbeat, Markdown memory, skills.
- OpenClaw vulnerability research: prompt injection and malicious skills.
Competitors, as of October 28, 2026
- Arga Labs: hosted; 46 twins; $10M seed. Its GitHub twin works with
gh,gitand GraphQL, but uncovered calls return silent 200s, and customers grade with browser assertions only. - Archal: hosted; 22 environments with hash-pinned samples. It removed its LLM-judge grading, and its paid tier is not yet open.
- Rystic: local Go simulators for 17 APIs with published match rates. It runs one process per tool with no cross-tool reactions, and has just added pass/fail grading.