Policy
Maps an observation and a language instruction to an action chunk, a horizon of actions executed open-loop as π0, ACT, and diffusion policies do.
The framework
One framework, two swappable inputs
LLM evals have a single swappable input: the model. Robotics evals have two, and Inspect Robots makes both first-class and orthogonal.
Embodiment
Produces observations, executes actions, and owns the action and observation spaces and control rate. Real robots come first; simulators are a stricter special case.
A Task, a dataset of Scenes plus scorers, is defined independently of both. Before any rollout, Inspect Robots verifies the policy and embodiment are compatible and fails fast if they are not.
Try it
Quickstart
No hardware or simulator required. The CubePick mock world exercises the whole stack.
from inspect_robots import eval
from inspect_robots.mock import CubePickEmbodiment, ScriptedPolicy
from inspect_robots.scene import Scene
from inspect_robots.scorer import success_at_end
from inspect_robots.task import Task
task = Task(
name="cubepick-reach",
scenes=[Scene(id=f"layout-{i}", instruction="reach the cube", init_seed=i) for i in range(5)],
scorer=success_at_end(),
max_steps=80,
)
# The two swappable inputs: a policy (VLA) and an embodiment (robot/sim).
(log,) = eval(task, ScriptedPolicy(), CubePickEmbodiment())
print(log.status, log.results.metrics) # success {'success_at_end': 1.0}
Design
Why Inspect Robots
Real-world first
Interfaces assume real-robot reality: human-in-the-loop reset, no privileged success oracle, and a wall-clock control rate. Simulators just offer more (seeding, privileged success, rendering) via opt-in capabilities.
Reproducible
Every run yields an immutable, schema-versioned EvalLog with the resolved config, git revision, and package versions. It is re-readable across releases and re-scorable offline.
Light core
Depends only on NumPy. Rerun and simulator or VLA backends are optional extras and separately installable plugins.
Safe unattended
An explicit error taxonomy separates record and continue from halt and require a human, so a faulted robot never auto-advances overnight.
Rerun visualization
Stream camera images, 3D poses, joint and action time-series, and success markers to a .rrd recording. Logging is non-blocking: a slow viewer connection drops camera frames first instead of delaying the robot control loop.
Pluggable
Backends ship as separate packages: the first-party plugins below, and rig plugins like inspect-robots-yam. Entry points make them appear in inspect-robots list automatically.
For Inspect AI users
How it maps to Inspect AI
If you know Inspect AI, you already know Inspect Robots.
| Inspect AI | Inspect Robots |
|---|---|
Model | Policy (VLA) + Embodiment (two inputs) |
Task = dataset + solver + scorer | Task = scenes + controller + scorer |
Sample | Scene |
Solver chain | Controller middleware (chunking, ensembling, smoothing) |
eval() → EvalLog | eval() → EvalLog |
@task / @solver / @scorer + registry | @task / @policy / @embodiment / @scorer + entry points |
Ecosystem
First-party plugins
Both halves of an eval, the body and the brain, have ready-made adapters shipped from this repository as separate packages.
inspect-robots-rosRun evals on ROS 1 or ROS 2 arms through rosbridge, with no ROS installation on the eval machine (--embodiment ros).inspect-robots-isaacsimRun evals against an Isaac Lab simulation (--embodiment isaacsim).inspect-robots-xpolicylabDrive any XPolicyLab-served policy. One adapter puts its zoo of 40+ VLAs (π0/π0.5, GR00T, OpenVLA-OFT, RDT-1B, SmolVLA, ACT, …) behind --policy xpolicylab -P url=ws://gpu-box:19000.inspect-robots-agentLet a frontier LLM (Claude, GPT, anything behind an OpenAI-compatible API) drive any embodiment through tool calls, as a first-class policy. The same --policy agent runs ad-hoc instructions and scores on registered tasks next to fine-tuned VLAs.inspect-robots-capxEvaluate CaP-X-style code-as-policy agents against a joint-space embodiment. Model-generated Python calls separately served SAM3, Contact-GraspNet, and Pyroki helpers, then queues approver-checked joint targets behind --policy capx.