A Arena

Reference

WebMCP Evals compatibility.

Use GoogleChromeLabs webmcp-evals for live model and browser execution, then use Arena to independently check the imported trajectory, Chrome guidance, and signed behavioral evidence.

Why both tools?

webmcp-evals answers whether a model selected and called the expected tools. Arena treats that report as untrusted input, recomputes every expected trajectory, audits the tool and runtime observations, then verifies an Arena boundary proof. A claimed upstream pass count cannot override a mismatched call.

Command

npx arena eval \
  --evals examples/webmcp-evals/evals.json \
  --results examples/webmcp-evals/results.json \
  --tools examples/webmcp-evals/tools.json \
  --observations examples/webmcp-evals/observations.json \
  --proof /path/to/arena-proof.json \
  --format json

Without --proof, Arena returns inconclusive. It never converts tool-selection success into a behavioral safety claim.

Three independent layers

  1. Selection: matches ordered, unordered, optional, unconstrained, argument, and result expectations from the current eval format.
  2. Guidance: checks naming and description budgets, annotations, origin exposure, token limits, untrusted-content flow, confirmation, mid-chain failure, and cancellation.
  3. Behavior: verifies the signed Arena proof against authoritative effects. This is the only layer that establishes human-route protection parity.

Runtime observations

Observations must come from the adapter, agent harness, or authoritative backend trace. Do not populate them from a tool description. Validate the file against the observation schema.

Upstream references