# Inspect Robots

Inspect Robots is the open-source evaluation framework for physical AI and VLA (vision-language-action) models: the "Inspect AI for robotics". A benchmark is a Task (dataset of Scenes + scorers) run against two swappable inputs: a Policy (the VLA) and an Embodiment (a real robot or simulator).

---

<!-- Source: guide/quickstart.md -->

# Quickstart

## Install

In a fresh directory (or your existing project), create a virtual environment
and install Inspect Robots with Rerun visualization:

```bash
uv venv && uv pip install "inspect-robots[rerun]"
```

For the NumPy-only core:

```bash
uv venv && uv pip install inspect-robots
```

Any virtual environment workflow works. Activate it once (`source
.venv/bin/activate`; `.venv\Scripts\activate` on Windows) and call
`inspect-robots` directly.

:::note
Invoke the CLI as plain `inspect-robots`, not `uv run inspect-robots`. Inside a
uv project, `uv run` re-syncs to the lockfile and can uninstall what `uv pip
install` just added.
:::

## Run your first evaluation

The dependency-free `CubePick` mock world lets you exercise the whole stack with
no hardware or simulator:

```python
from inspect_robots import eval
from inspect_robots.mock import CubePickEmbodiment, ScriptedPolicy
from inspect_robots.scene import Scene
from inspect_robots.scorer import success_at_end
from inspect_robots.task import Task

task = Task(
    name="cubepick-reach",
    scenes=[Scene(id=f"layout-{i}", instruction="reach the cube", init_seed=i) for i in range(5)],
    scorer=success_at_end(),
    max_steps=80,
)

(log,) = eval(task, ScriptedPolicy(), CubePickEmbodiment())
print(log.status)                    # "success"
print(log.results.metrics)           # {"success_at_end": 1.0}
```

`eval()` returns a list of [`EvalLog`](/api/#inspect_robots.log.EvalLog) (one per task, mirroring
Inspect AI). Each log is immutable, schema-versioned, and written to `log_dir`.

## Use registry names

`task`, `policy`, and `embodiment` may also be registry names, the same
mechanism the CLI uses:

```python
from inspect_robots import eval

(log,) = eval("cubepick-reach", "scripted", "cubepick")
```

## From the command line

The CLI resolves any registered task, policy, or embodiment (builtins plus
installed plugins):

```bash
inspect-robots list                                          # all registered components
inspect-robots list policies                                 # just policies
inspect-robots run --task cubepick-reach --policy scripted --embodiment cubepick
inspect-robots run --task cubepick-reach --policy scripted --embodiment cubepick -P chunk_size=6
inspect-robots inspect logs/cubepick-reach_*.json            # print a saved log
inspect-robots view logs/cubepick-reach_*.json               # render an HTML report
inspect-robots video logs/cubepick-reach_*.json              # camera frames to MP4
```

`view` writes a self-contained HTML page. `video` needs a run captured with
`--store-frames` and the `ffmpeg` binary on PATH. Every command is covered in
[the CLI guide](cli.md).

## On a real robot

Install the plugin for your rig and set your defaults once:

```bash
source .venv/bin/activate
uv pip install inspect-robots-yam   # provides the molmoact2 policy + yam_arms rig
inspect-robots setup
```

The wizard picks your defaults and finds your cameras, then writes
`~/.config/inspect-robots/config.ini`. On a different rig, install its plugin
instead and type its component names at the prompts; to write the config file
by hand, see [the CLI guide](cli.md).

The `molmoact2` policy is only a client: nothing moves until the MolmoAct2
server is listening, and the server does not start itself or survive a reboot
(full setup in the
[yam plugin README](https://github.com/robocurve/inspect-robots-yam#install-on-the-robotgpu-machine)):

```bash
# On the GPU machine, from the MolmoAct2 repo. Leave it running, e.g. in tmux:
python examples/yam/host_server_yam.py --host 0.0.0.0 --port 8202
curl http://127.0.0.1:8202/act      # 200 means the server is ready
```

On a different rig, start whatever serves your policy instead; in-process
policies (such as `agent` or the mock `scripted`) need no server.

Then tell the robot what to do:

```bash
inspect-robots "place the fork on the plate"
```

Every run opens a live Rerun viewer streaming the cameras, proprioception,
and actions straight from the eval pipeline, so you watch exactly what the
policy sees while the robot moves. CLI flags override any default
(`--no-rerun`, `--no-store-frames`, `--max-steps 300`, ...).

Safety guardrails (a bounds clamp plus a per-step delta limit derived from
the embodiment's action space) are wired into every CLI run by default, for
every policy. Turning them off requires an explicit `--disable-guardrails`.

## Drive the robot with an LLM

The policy slot is not limited to VLAs. With the
[inspect-robots-agent](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-agent)
plugin, a frontier LLM drives the same rig through tool calls, one
approver-checked motion chunk per call.

Put a `.env` with your API key in the working directory (the CLI loads it
automatically):

```ini
ANTHROPIC_API_KEY=sk-ant-...
```

Install the add-on:

```bash
uv pip install inspect-robots-agent
```

Run the LLM on the robot:

```bash
inspect-robots "place the fork on the plate" --policy agent \
    -P model=anthropic/claude-fable-5 -P effort=low
```

No rig? The same policy drives the mock world, no hardware or GPU required
(an exported key works in place of `.env`):

```bash
export ANTHROPIC_API_KEY=sk-ant-...
inspect-robots "pick up the cube" --policy agent \
    -P model=anthropic/claude-fable-5 -P effort=low --embodiment cubepick
```

Read the recorded agent conversation with
`inspect-robots inspect LOG.json --transcript`, or open the HTML report with
`inspect-robots view LOG.json`; for `--store-frames` runs it includes the
camera frames the model saw.

## Generate robot policy code with CaP-X

The
[inspect-robots-capx](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-capx)
plugin evaluates a code-as-policy agent in the same policy slot. The LLM
writes Python against SAM3 segmentation, Contact-GraspNet planning, Pyroki
IK, and speed-limited joint-motion helpers.

```bash
uv pip install inspect-robots-capx

inspect-robots "place the fork on the plate" --policy capx \
    --embodiment <joint-space-embodiment> \
    -P model=anthropic/claude-fable-5 -P sam3_url=http://gpu-box:8114
```

See the plugin README for embodiment requirements, model-server bringup, and
the model-code trust boundary.

## Run in simulation

The same instruction runs on your configured simulator instead of the real
robot (the mapping is explained in [the CLI guide](cli.md)):

```bash
inspect-robots "place the fork on the plate" --sim
```

## Next steps

- [Concepts](concepts.md): the core abstractions.
- [Writing a benchmark](writing-a-benchmark.md): define your own `Task`.
- [Policies and embodiments](policies-and-embodiments.md): plug in a real VLA or robot/sim.
- [Plugins](plugins.md): the first-party adapters for ROS, Isaac Lab, XPolicyLab, LLM agents, and CaP-X.
- [The CLI](cli.md): configuration, `--sim` mapping, scoring, and every command.

---

<!-- Source: guide/concepts.md -->

# Concepts

Inspect Robots factors a robotics evaluation into a few small, orthogonal pieces.

## The two inputs

Unlike LLM evals (one swappable input, the model), a robotics eval has two:

- [`Policy`](/api/#inspect_robots.policy.Policy): the VLA "brain". Given an
  [`Observation`](/api/#inspect_robots.types.Observation), returns an
  [`ActionChunk`](/api/#inspect_robots.types.ActionChunk): a horizon of actions executed open-loop
  (because VLA inference is slower than the control rate). `H = 1` is the
  degenerate reactive case.
- [`Embodiment`](/api/#inspect_robots.embodiment.Embodiment): the "body + world" (a real robot or
  a simulator). It produces observations, executes actions, and owns the
  action/observation spaces, the native control rate, and reset/safety machinery.

Both are runtime-checkable Protocols, so you can wrap an existing model or sim
without inheriting anything. Convenience base classes (`PolicyBase`,
`EmbodimentBase`) exist if you prefer.

## Tasks and scenes

A [`Task`](/api/#inspect_robots.task.Task) is an embodiment-agnostic benchmark: a dataset
of [`Scene`](/api/#inspect_robots.scene.Scene)s plus scorer(s), a step- or seconds-based
horizon, and an epoch count. A seconds horizon is resolved to integer steps from
the paired embodiment's declared control rate. A `Scene` is the robotics analog of Inspect AI's `Sample`, one initial
condition: an instruction, an optional success [`Target`](/api/#inspect_robots.scene.Target),
and a seed.

## Compatibility

Before any rollout, [`check_compatibility`](/api/#inspect_robots.compat.check_compatibility) verifies the
`(policy, embodiment)` pair: action dimensions and [`ActionSemantics`](/api/#inspect_robots.spaces.ActionSemantics)
(control mode, rotation representation, gripper, frame), the observation
cameras/state keys the policy requires (resolving a name remap), the control rate,
whether a seconds-based task has a usable control rate, and whether each scene
is realizable on the embodiment. Hard mismatches fail fast
with a [`CompatibilityError`](/api/#inspect_robots.errors.CompatibilityError).

## The rollout

[`rollout`](/api/#inspect_robots.rollout.rollout) runs one trial as a single control-rate loop:

1. A [`Controller`](/api/#inspect_robots.controller.Controller) decides the next action, internally
   calling `policy.act()` and buffering the chunk (so open-loop execution and
   temporal ensembling compose without forking the loop).
2. An [`Approver`](/api/#inspect_robots.approver.Approver) reviews the action before it reaches
   the embodiment: pass, clamp, or veto (a safety gate).
3. `embodiment.step(action)` executes it; everything is logged to sinks and
   recorded in an immutable [`TrialRecord`](/api/#inspect_robots.rollout.TrialRecord) (steps, a typed
   transcript, inference latencies).

Camera frames are streamed to a [`FrameStore`](/api/#inspect_robots.frames.FrameStore) and the
record keeps lightweight references, so long multi-camera episodes stay
memory-safe.

## Scoring

A [`Scorer`](/api/#inspect_robots.scorer.Scorer) maps a recorded `TrialRecord` (+ the scene's
`Target`) to a [`Score`](/api/#inspect_robots.scorer.Score). Because scorers consume the
*recorded* trajectory (not a live environment), scoring is reproducible from a
saved log. Across the `epochs` of a scene, an epoch reducer (`mean`, `max`,
`pass_at_k`, …) collapses scores; metrics then aggregate across scenes.

## Errors and safety

The error taxonomy resolves the "fail fast vs never-crash-overnight" tension:

| Class | Policy |
|---|---|
| [`CompatibilityError`](/api/#inspect_robots.errors.CompatibilityError), `ConfigError` | fail fast, before any rollout |
| [`PolicyError`](/api/#inspect_robots.errors.PolicyError) | record the trial, then continue or halt per `fail_on_error` (`True` = first error, `0<x<1` = proportion, `x>1` = count), checked after every trial |
| [`EmbodimentFault`](/api/#inspect_robots.errors.EmbodimentFault), [`SafetyAbort`](/api/#inspect_robots.errors.SafetyAbort) | **always halt**: a faulted/unsafe robot never auto-advances |

Failures inside a trial (including `reset`) are wrapped into the taxonomy; a
crashing approver becomes a `SafetyAbort` (it can no longer vouch for safety).
Every error raised from inside a trial carries the partial
[`TrialRecord`](/api/#inspect_robots.rollout.TrialRecord) on its `record` attribute, so the
steps that did run are delivered to sinks. Errored trials are recorded but
never scored: a failed trial cannot masquerade as data in the metrics.
A finished run in which every trial errored ends with `status: "error"` even
under the default `fail_on_error=False`: a run that scored nothing is not a
success.

## The eval log

[`eval`](/api/#inspect_robots.eval.eval) orchestrates scenes × epochs and returns immutable
[`EvalLog`](/api/#inspect_robots.log.EvalLog)s (status, spec, results, stats, per-scene samples,
error). Logs are written atomically as schema-versioned JSON with a read-back
guarantee. Once rollouts have started, an `EvalLog` is always produced and
persisted: scorer or reducer failures degrade the run to an error log rather
than crashing away the night's data.

`eval()` also owns what it opens: an embodiment resolved from a registry
name is closed when the run finishes (even on a halt), while a
caller-constructed embodiment object stays open (its lifecycle belongs to the
caller).

Passing `seed=None` draws a fresh seed from the OS and records it in the log,
so an "unseeded" run remains reproducible after the fact (and is distinct from
`seed=0`).

---

<!-- Source: guide/writing-a-benchmark.md -->

# Writing a benchmark

A benchmark is a [`Task`](/api/#inspect_robots.task.Task): a dataset of scenes plus scorer(s).
It is embodiment-agnostic: it describes *what* to evaluate, not *how* the
robot is built.

```python
from inspect_robots.scene import Scene, Target
from inspect_robots.scorer import success_at_end
from inspect_robots.task import Epochs, Task

task = Task(
    name="cubepick-reach",
    scenes=[
        Scene(
            id=f"layout-{i}",
            instruction="reach the cube",
            target=Target(kind="reach_object", spec={"object": "cube"}),
            init_seed=i,
        )
        for i in range(50)
    ],
    scorer=success_at_end(),
    max_steps=200,
    epochs=Epochs(count=3, reducer="mean"),
)
```

## Scenes

Each [`Scene`](/api/#inspect_robots.scene.Scene) is one initial condition (the Inspect `Sample`
analog):

- `id`: unique within the task.
- `instruction`: the language goal handed to the policy.
- `target`: an optional [`Target`](/api/#inspect_robots.scene.Target) the scorer reads; its
  `kind` is resolved in the *embodiment's* namespace (compatibility checking
  verifies the embodiment can realize it).
- `init_seed`: combined with the eval seed and epoch index to seed each trial
  deterministically. (An eval run with `seed=None` draws a fresh OS seed and
  records it in the log, so even "unseeded" runs are reproducible after the
  fact, and distinct from `seed=0`.)

## Epochs and reducers

Repeat each scene `epochs` times to measure stochastic policies. The
[`Epochs`](/api/#inspect_robots.task.Epochs) reducer collapses the per-epoch scores of a scene
before metrics aggregate across scenes. Builtin reducers: `mean`, `median`,
`max`, `min`, `mode`, and `pass_at_<k>` (an unbiased pass@k estimator).

## Multiple scorers

Pass a list to score several dimensions at once:

```python
from inspect_robots.scorer import episode_length, min_distance_to_goal, success_at_end

task = Task(
    name="cubepick-reach",
    scenes=[...],
    scorer=[success_at_end(), episode_length(), min_distance_to_goal()],
    max_steps=200,
)
```

## Horizons

A task declares exactly one rollout horizon. Use `max_steps` when the protocol
is inherently discrete, as in the examples above. Use `max_seconds` when every
embodiment should receive the same physical-time budget:

```python
task = Task(
    name="two-minute-reach",
    scenes=[...],
    scorer=success_at_end(),
    max_seconds=120.0,
)
```

At evaluation time, Inspect Robots resolves the budget as
`ceil(max_seconds * embodiment.info.control_hz)`. A 120-second task therefore
runs for 1,200 steps at 10 Hz and 1,800 steps at 15 Hz. The eval log records
both the declared seconds and the resolved integer step limit.

A seconds-based task is incompatible with an embodiment whose `control_hz` is
missing, non-finite, zero, or negative. Event-driven embodiments should use
`max_steps`. Resolution changes the step budget only: `rollout()` does not add
wall-clock pacing, so real-time cadence remains the embodiment's responsibility.

## Registering for discovery

Wrap a task factory with [`task`](/api/#inspect_robots.registry.task) so it resolves by name in
`eval("my-bench", ...)` and appears in `inspect-robots list`:

```python
from inspect_robots.registry import task

@task("my-bench")
def my_bench(num_scenes: int = 50) -> Task:
    return Task(name="my-bench", scenes=[...], scorer=success_at_end(), max_steps=200)
```

See [Plugins](plugins.md) to ship a benchmark from a separate package.

---

<!-- Source: guide/policies-and-embodiments.md -->

# Policies and embodiments

Both are runtime-checkable Protocols: implement the methods on any class (no
inheritance required), or subclass the convenience base classes.

## A policy (VLA)

A [`Policy`](/api/#inspect_robots.policy.Policy) maps an observation to an
[`ActionChunk`](/api/#inspect_robots.types.ActionChunk). It declares a
[`PolicyInfo`](/api/#inspect_robots.policy.PolicyInfo) (the action space it emits and the
observations it requires) used for compatibility checking.

```python
import numpy as np
from inspect_robots.policy import PolicyConfig, PolicyInfo
from inspect_robots.scene import Scene
from inspect_robots.spaces import ActionSemantics, Box, ObservationSpace
from inspect_robots.types import Action, ActionChunk, Observation


class MyVLA:
    def __init__(self) -> None:
        self.info = PolicyInfo(
            name="my-vla",
            action_space=Box(
                shape=(7,),
                semantics=ActionSemantics(
                    control_mode="eef_delta_pose", rotation_repr="euler_xyz", gripper="continuous"
                ),
            ),
            observation_space=ObservationSpace(
                state_keys=frozenset({"eef_pose", "gripper"}),
            ),
        )
        self.config = PolicyConfig(action_horizon=16)

    def reset(self, scene: Scene) -> None:
        ...  # clear any per-episode state

    def act(self, observation: Observation) -> ActionChunk:
        # The policy owns model-specific preprocessing (resize/normalize/history).
        chunk = my_model_infer(observation)        # -> (H, 7) array
        actions = [Action(data=a) for a in chunk]
        return ActionChunk(actions=actions, inference_latency_s=...)
```

The policy owns model-specific spatial preprocessing; the embodiment emits raw
frames. Temporal concerns (history, smoothing, ensembling) live in a
[Controller](../guide/concepts.md).

## An embodiment (robot or sim)

An [`Embodiment`](/api/#inspect_robots.embodiment.Embodiment) produces observations and executes
actions. It declares an [`EmbodimentInfo`](/api/#inspect_robots.embodiment.EmbodimentInfo) with its
spaces, native control rate, and opt-in capability flags.

```python
from inspect_robots.embodiment import EmbodimentInfo, PRIVILEGED_SUCCESS, SEEDABLE
from inspect_robots.scene import Scene
from inspect_robots.spaces import Box, CameraSpec, ObservationSpace
from inspect_robots.types import Action, Observation, StepResult


class MyArm:
    def __init__(self) -> None:
        self.info = EmbodimentInfo(
            name="my-arm",
            action_space=Box(shape=(7,), semantics=...),
            observation_space=ObservationSpace(
                cameras=(CameraSpec("base_rgb", 224, 224), CameraSpec("wrist_rgb", 224, 224)),
                state_keys=frozenset({"eef_pose", "gripper"}),
            ),
            control_hz=20.0,
            is_simulated=False,
            capabilities=frozenset({SEEDABLE}),  # real arms rarely have PRIVILEGED_SUCCESS
        )

    def reset(self, scene: Scene, *, seed: int | None = None) -> Observation:
        # On real hardware this may drive to home and block on operator confirmation.
        ...

    def step(self, action: Action) -> StepResult:
        # The rollout applies no wall-clock pacing of its own. If this embodiment
        # needs real-time cadence, pace it here and declare "self_paced".
        ...

    def close(self) -> None:
        ...
```

When a human operator ends an episode without giving a verdict (an end-episode
keypress), terminate with `termination_reason=inspect_robots.OPERATOR_END`
(`"operator_end"`). Attended runs (interactive terminal, no `--no-prompt`) then
get the operator prompt — `did the robot succeed? [y/n/partial/skip]` plus an
optional grader note — for exactly those trials, on registered tasks and ad-hoc
runs alike. Do **not** ask your own verdict prompt in the embodiment and
terminate with a definitive `"success"`/`"failure"`: that suppresses the
framework prompt and locks the run out of partial/skip verdicts and notes.

Lifecycle: [`eval`](/api/#inspect_robots.eval.eval) closes what it resolves. An
embodiment looked up by registry name is closed when the run finishes (even
on a halt). If you construct the embodiment object yourself, you own it: call
`close()` yourself when you are done.

## Real-robot vs simulator

The interfaces assume real-robot reality: no guaranteed privileged success,
human-in-the-loop reset, wall-clock control. Simulators opt into more via
`capabilities` (`SEEDABLE`, `AUTO_RESET`, `PRIVILEGED_SUCCESS`, `RENDERABLE`, …).
A sim may put privileged success into `StepResult.info` for a scorer to read; a
real robot typically relies on an operator verdict
([`operator_scorer`](/api/#inspect_robots.scorer.operator_scorer)) or a learned classifier.

## Compatibility

If the policy's action dimension/semantics or required observations don't match
the embodiment, [`eval`](/api/#inspect_robots.eval.eval) raises a
[`CompatibilityError`](/api/#inspect_robots.errors.CompatibilityError) before any rollout. Use `remap=` to
alias differing camera/state key names:

```python
eval(task, MyVLA(), MyArm(), remap={"base_rgb": "camera_0"})
```

---

<!-- Source: guide/scoring.md -->

# Scoring

A [`Scorer`](/api/#inspect_robots.scorer.Scorer) maps a recorded
[`TrialRecord`](/api/#inspect_robots.rollout.TrialRecord) (plus the scene's
[`Target`](/api/#inspect_robots.scene.Target)) to a [`Score`](/api/#inspect_robots.scorer.Score). Scorers
read the *recorded* trajectory (never a live environment), so scoring is
reproducible from a saved log.

## Builtin scorers

```python
from inspect_robots.scorer import (
    success_at_end,        # 1.0 iff the episode terminated with reason "success"
    episode_length,        # number of steps taken
    min_distance_to_goal,  # closest the effector got (reads StepResult.info["distance"])
    reached_goal_state,    # success iff min distance <= threshold
    operator_scorer,       # reads a human verdict recorded during the rollout
)
```

## Custom scorers

A scorer is any object with a `name` and a `__call__(record, target) -> Score`:

```python
from dataclasses import dataclass
from inspect_robots.scorer import Score

@dataclass(frozen=True)
class SmoothMotion:
    name: str = "smooth_motion"

    def __call__(self, record, target) -> Score:
        deltas = [abs(float(s.action.data.sum())) for s in record.steps]
        return Score(value=-sum(deltas), explanation="negative total command magnitude")
```

Register it with [`scorer`](/api/#inspect_robots.registry.scorer) to resolve it by name.

## Epochs and reducers

When a `Task` runs `epochs > 1`, an epoch reducer collapses the per-epoch
scores of a scene before metrics aggregate across scenes. Reducers are namespaced
separately from metrics and are selected by name on
[`Epochs`](/api/#inspect_robots.task.Epochs):

| Reducer | Meaning |
|---|---|
| `mean`, `median`, `max`, `min` | numeric reductions (raise on non-numeric strings) |
| `mode` | most common value (works for categorical scores) |
| `pass_at_<k>` | unbiased pass@k estimator (success = value ≥ 0.5) |

```python
from inspect_robots.task import Epochs, Task
Task(..., epochs=Epochs(count=5, reducer="pass_at_2"))
```

## Operator and VLM scoring (real world)

Real robots have no privileged success oracle. The dominant method is a human
verdict, captured *once* during the rollout (as a transcript event) and read
back by [`operator_scorer`](/api/#inspect_robots.scorer.operator_scorer), keeping scoring reproducible.
A [`VLMScorer`](/api/#inspect_robots.scorer.VLMScorer) interface is reserved for scoring final
frames with a vision-language classifier.

Trials that end with `termination_reason="operator_end"` are prompted on any
attended run — registered tasks included — so judgement-reading scorers (the
`operator` scorer, or task scorers that fall back to `operator_judgement`)
work with operator-in-the-loop embodiments. `success_at_end` reads only
embodiment-detected `"success"` terminations and scores operator-graded trials
as failures; pair attended operator-graded runs with a judgement-reading
scorer instead.

---

<!-- Source: guide/logging-and-rerun.md -->

# Logging & Rerun

## The eval log

Every run produces an immutable [`EvalLog`](/api/#inspect_robots.log.EvalLog): the canonical,
auditable record. It mirrors Inspect AI: `version`, `status`, an `eval` spec
(task/policy/embodiment, created time, git revision, package versions), `results`
(aggregate metrics), `stats` (timing, inference latency), per-scene `samples`, and
a structured `error`.

```python
from inspect_robots import eval, read_eval_log

(log,) = eval("cubepick-reach", "scripted", "cubepick", log_dir="logs")
again = read_eval_log("logs/cubepick-reach_xxxx.json")   # always re-readable
```

Logs are written atomically (temp file + rename), schema-versioned, and carry
a read-back guarantee: a newer Inspect Robots always reads an older log.

Pressing Ctrl-C during a rollout writes a log with `status: "cancelled"` and
everything gathered so far, including the partial trial record and transcript.

## Sinks

A [`LogSink`](/api/#inspect_robots.logging.sink.LogSink) observes the run lifecycle
(`on_eval_start` → per trial `on_trial_start`/`log_step`/`on_trial_end` →
`on_eval_end`). Builtins:

- [`JsonLogSink`](/api/#inspect_robots.logging.json_log.JsonLogSink): the default; the canonical JSON record.
- [`RerunSink`](/api/#inspect_robots.logging.rerun_sink.RerunSink): optional, lazily imported.

Passing `sinks=` replaces the default `JsonLogSink`, it does not add to it.
Include one in the list if you still want the JSON log:

```python
from inspect_robots.logging import JsonLogSink, RerunSink

eval(task, policy, embodiment, sinks=[JsonLogSink("logs"), RerunSink("run.rrd")])
```

## Rerun visualization

`RerunSink` streams camera images, proprioception, action vectors, reward, and
termination markers to a [Rerun](https://github.com/rerun-io/rerun) recording. It
imports `rerun-sdk` lazily: if it isn't installed, the sink warns once and
no-ops, so core never depends on it. Install with `pip install "inspect-robots[rerun]"`.

The sink lays out labeled joint series per arm, with commanded `action/*` and
measured `state/*` together for each side and cameras, the LLM transcript, and
reward alongside. Embodiments without `dim_labels` get one combined joints
plot. The layout is re-sent at each trial boundary so it follows the live trial,
which resets viewer tweaks then; a single-trial `run` sends it exactly once.
The layout is built from the declared spaces, so entities logged outside them
(an undeclared state key, or a camera whose runtime name differs from its
declared name) are not shown by the sent layout and must be added in the
viewer manually.

Logging is non-blocking. `log_step` snapshots each transition and a background
worker hands it to the SDK, so a slow or stalled viewer connection never delays
the control loop (on real hardware, a blocked viewer used to stall the robot
mid-episode). Under sustained backpressure the sink degrades visualization
instead of control: camera frames are dropped first, so scalar plots stay
complete, then whole steps, and the totals are reported as a `RuntimeWarning`
when the eval ends. The queue is drained at every trial boundary (bounded by
`flush_timeout`), so an eval that aborts mid-run loses at most the current
trial's queued tail; the JSON eval log is synchronous and never affected.

Camera frames are JPEG-compressed by default (`jpeg_quality=75`), which cuts
viewer bandwidth by an order of magnitude. Pass `jpeg_quality=None` for
pixel-exact frames. Compression needs pillow (the `rerun` extra includes it);
without it the sink warns once and logs raw frames. Frames of record are never
at stake either way: scoring reads from the `FrameStore` side-car, not from
Rerun.

```python
RerunSink("run.rrd")                   # record to a file, view later
RerunSink(spawn=True)                  # live viewer on this machine (CLI: --rerun)
RerunSink(connect_url="rerun+http://127.0.0.1:9876/proxy")  # stream to a running viewer
RerunSink(spawn=True, jpeg_quality=None, queue_size=128)  # lossless, deeper buffer
```

The three modes are mutually exclusive: rerun's `save`/`spawn`/`connect_grpc`
calls each replace the SDK's global sink, so combining them raises `ValueError`
rather than silently dropping a stream.

### Live transcript in the viewer

Policies that support transcript streaming automatically add conversation rows
at `trial/<scene>/e<epoch>/llm`. In the Rerun viewer, add a TextLog view and
select that entity path. Tool results use the DEBUG level and system prompts use
TRACE, so enable both levels in the view's log-level filter to see the whole
conversation. The most recent assistant message is also available as markdown
at `trial/<scene>/e<epoch>/llm/latest`; add a Text Document view for a wrapped
reading pane that stays synchronized with the timeline cursor and always shows
the policy's latest decision.

Scrubbing the `step` timeline highlights the transcript rows emitted for that
control step alongside its camera and state data. This live stream is a
best-effort visualization and transcript updates may be dropped under
backpressure. The transcript persisted in the eval log is collected separately
at trial end and remains the complete audit record.

On a headless robot box, `spawn=True` has nowhere to open a window. Run the
viewer on your own machine instead and stream to it: `rerun` on your laptop,
`ssh -R 9876:localhost:9876 <robot>` for the tunnel, then
`inspect-robots run ... --rerun-connect` (a bare `--rerun-connect` targets the
tunnel's localhost URL above; pass a URL to reach a viewer elsewhere). Viewer
and SDK versions must match for live connections.

## Frame side-cars

Camera frames are large. With `store_frames=True`, the rollout streams frames to
a per-run subdirectory of `<log_dir>/frames` through a
[`FrameStore`](/api/#inspect_robots.frames.FrameStore) and the `TrialRecord` keeps lightweight
[`FrameRef`](/api/#inspect_robots.frames.FrameRef) handles, so long, multi-camera episodes stay
memory-safe and remain scorable from disk. Trial ids repeat across runs, so
each eval gets its own directory; read the exact path from the log's
`stats.frames_dir` rather than globbing `<log_dir>/frames` directly.

```python
eval(task, policy, embodiment, log_dir="logs", store_frames=True)
```

Stored frames are raw `.npy` arrays, not a video. To watch an episode after
the fact, render them with the [`video` subcommand](cli.md#inspect-robots-video):

```bash
inspect-robots video logs/adhoc_xxxx.json
```

`inspect-robots inspect` prints the frames directory and this command as a
hint whenever a log has stored frames.

## Policy transcripts

Policies can persist a per-trial audit record in the eval log; read it with
`inspect-robots inspect LOG.json --transcript`, or render a self-contained
conversation page with [`inspect-robots view`](cli.md#inspect-robots-view):

```bash
inspect-robots view LOG.json
```

The agent policy stores its conversation, with streamed image bytes replaced by
`[image omitted: streamed camera frame]`. The preceding label, such as
`camera 'top_cam' (step 480):`, is emitted whether or not frames are stored,
and when they are (`store_frames=True`) it provides the step join key from a
transcript observation to the stored frame. `inspect-robots view` performs
this step join internally, embedding only an exact match and otherwise leaving
the placeholder in place.

`FrameStore` sanitizes trial and camera names before building
`{trial}_{camera}_{t:06d}.npy`. When the sanitizer rewrites a name, use
`StepRecord.image_refs` and `FrameRef.path` as the authoritative mapping instead
of assembling the path from the transcript label. That remains the right advice
for programmatic consumers. The `view` command performs this join internally
with the same sanitizer and an exact-match-or-degrade contract.

## Wire capture

The agent policy records **exactly what each LLM call sent and received** —
by default, for every run. The saved transcript alone is not that object:
outgoing requests carry the tool schemas, only the newest `image_horizon`
frames (older ones become elision stubs), rendered depth composites, and, on
the Anthropic wire, `cache_control` breakpoints — none of which survive into
`policy_transcripts`. Wire capture stores the real thing, per attempt,
including retries and the failed calls a run died on.

Layout, under the log directory:

```
wire/<run_id>/<trial_id>/calls.jsonl   one JSON row per HTTP attempt
wire/<run_id>/blobs/<sha256>.png       content-addressed image bytes
```

Each row records `call`, `attempt`, `endpoint`, `t`, `duration_s`,
`request`, `status`, `response`, and (failed attempts only) `error`. Image
payloads inside `request` are replaced by `$blob:<sha256>` sentinels — the
sha of the decoded PNG stored once in `blobs/` — with every other part key,
including data-URL prefixes and `cache_control`, preserved verbatim.
Restoring a byte-faithful request is a string substitution: replace each
sentinel with the base64 of its blob file. The trial's record points at its
capture via `metadata["wire_capture"]`.

Browse it with either viewer:

- `inspect-robots view <log>` — each trial gains a collapsible **Wire**
  section: per-call status, params, delta-rendered messages as sent, and
  the frames the model could actually see, deduplicated against the
  report's embedded-media budget.
- `inspect-robots inspect <log> --wire` — a per-trial call table;
  `--wire N` (with `--trial <scene>-e<epoch>` when the log has several
  trials) dumps every attempt row of call `N`.

Capture is best-effort and can never fail an eval: any sink error prints
one warning and disables capture for that trial. Disable it entirely with
`-P wire_capture=false`. Budget for roughly `store_frames`-scale disk usage
(a 100-call three-camera trial with depth rendering writes on the order of
100 MB, dominated by blobs).

---

<!-- Source: guide/plugins.md -->

# Plugins & the registry

Inspect Robots components register by name and resolve from strings: the mechanism the
CLI and `eval("...", "...", "...")` use. In-tree builtins register via decorators;
out-of-tree packages publish entry points, so an installed plugin appears in
`inspect-robots list` without being imported first.

## Decorators

```python
from inspect_robots.registry import embodiment, policy, scorer, task

@policy("my-vla")
class MyVLA: ...

@embodiment("my-arm")
class MyArm: ...

@scorer("smooth")
def smooth(): ...

@task("my-bench")
def my_bench(): ...
```

## Resolving

```python
from inspect_robots.registry import registered, resolve

registered("policy")          # {"scripted": ..., "random": ..., "my-vla": ...}
policy = resolve("policy", "my-vla", checkpoint="...")   # constructor kwargs forwarded
```

## Shipping an out-of-tree plugin

Publish entry points from your package's `pyproject.toml`:

```toml
[project.entry-points."inspect_robots.embodiments"]
maniskill = "inspect_robots_maniskill:ManiSkillEmbodiment"

[project.entry-points."inspect_robots.policies"]
openvla = "inspect_robots_openvla:OpenVLAPolicy"
```

Groups: `inspect_robots.tasks`, `inspect_robots.policies`, `inspect_robots.embodiments`,
`inspect_robots.scorers`, `inspect_robots.sinks`. After `pip install inspect-robots-maniskill`, it
shows up in `inspect-robots list` and resolves by name in `eval()` and the CLI.

This is how the ecosystem stays decoupled: this repository is the framework;
specific simulators, VLA weights, and benchmarks live in their own packages.

### Reading user defaults

Plugin CLIs can read the configuration written by `inspect-robots setup`
through `inspect_robots.defaults`. Constructor args belong to the component
named by their owner field — apply `embodiment_args` only when the owner is
the embodiment *your* plugin drives, or args recorded for another rig will
configure yours:

```python
import os

from inspect_robots.defaults import load_defaults

defaults = load_defaults(os.environ)
if defaults.embodiment_args_owner == "my_embodiment":
    args = defaults.embodiment_args
```

A plugin that ships several embodiments (or expects subclasses registered by
other packages) can resolve the owner with
`inspect_robots.registry.registered("embodiment")` and check the factory's
class instead of comparing names.

## First-party plugins

Five adapters ship from the Inspect Robots repository as separate packages,
covering both halves of an eval:

- [`inspect-robots-ros`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-ros):
  run evals on ROS 1 or ROS 2 arms through rosbridge, with no ROS installation
  on the eval machine (`--embodiment ros`).
- [`inspect-robots-isaacsim`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-isaacsim):
  run evals against an [Isaac Lab](https://isaac-sim.github.io/IsaacLab/)
  simulation (`--embodiment isaacsim`).
- [`inspect-robots-xpolicylab`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-xpolicylab):
  drive any [XPolicyLab](https://github.com/XPolicyLab/XPolicyLab)-served policy.
  One adapter puts its zoo of 40+ VLAs (π0/π0.5, GR00T, OpenVLA-OFT, RDT-1B,
  SmolVLA, ACT, …) behind `--policy xpolicylab -P url=ws://gpu-box:19000`.
- [`inspect-robots-agent`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-agent):
  let a frontier LLM (Claude, GPT, or anything behind an OpenAI-compatible API)
  drive any embodiment through tool calls as a first-class policy. The same
  `--policy agent` runs ad-hoc instructions and scores on registered tasks next
  to fine-tuned VLAs.
- [`inspect-robots-capx`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-capx):
  evaluate CaP-X-style code-as-policy agents against a joint-space embodiment.
  Model-generated Python calls separately served SAM3, Contact-GraspNet, and
  Pyroki helpers, then queues approver-checked joint targets behind `--policy
  capx`.

### `inspect-robots-isaacsim`: the body

Wraps an [Isaac Lab](https://isaac-sim.github.io/IsaacLab/) simulation as an
embodiment. Installing it makes `isaacsim` resolvable; only `reset()`/`step()`
need a working Isaac install (listing and compatibility checks run anywhere):

```bash
pip install inspect-robots-isaacsim
inspect-robots run --task my-task --policy my-vla --embodiment isaacsim \
    -E task_id=Isaac-Lift-Cube-Franka-v0
```

### `inspect-robots-xpolicylab`: the brain

Drives any [XPolicyLab](https://github.com/XPolicyLab/XPolicyLab)-served
policy. XPolicyLab wraps 40+ VLA / imitation-learning policies (π0/π0.5,
GR00T, OpenVLA-OFT, RDT-1B, SmolVLA, ACT, …) behind one websocket
policy-server contract; this adapter speaks that protocol directly, so the
whole zoo becomes evaluable without installing any model dependencies locally:

```bash
pip install inspect-robots-xpolicylab

# terminal 1 — serve a policy from an XPolicyLab checkout (its own env/machine)
cd XPolicyLab/policy/Pi_0 && bash setup_eval_policy_server.sh ... 19000 0.0.0.0

# terminal 2 — evaluate it
inspect-robots run --task my-task --policy xpolicylab --embodiment isaacsim \
    -P url=ws://gpu-box:19000 -P cameras=cam_head:base_rgb
```

See each plugin's linked README for its full configuration reference.

---

<!-- Source: guide/adapters.md -->

# Authoring an embodiment adapter

Every embodiment adapter (a package registering an `inspect_robots.embodiments`
entry point) declares its spaces through `EmbodimentInfo`. Two consumers make
those declarations load-bearing:

- The CLI's default safety guardrails derive a bounds clamp and a per-step
  delta limit from the action space.
- The LLM agent policy (`inspect-robots-agent`) builds its whole tool surface
  from the spaces at bind time: tool schemas from the bounds and labels, the
  motion strategy from the control mode, the proprioceptive reference from
  the `StateSpec`. It also injects `EmbodimentInfo.docs` (free-form operating
  notes: joint layout, positive directions, gripper polarity) into its system
  prompt, so an adapter that ships good notes measurably helps LLM policies.

An adapter with missing or dishonest declarations silently degrades both.
This page is the contract; the conformance kit makes most of it mechanical.

## Declare runtime requirements

Declare imports that cannot be expressed by package metadata on the registered
component factory:

```python
RUNTIME_REQUIREMENTS = {
    "i2rt": 'uv pip install "i2rt @ git+https://github.com/i2rt-robotics/i2rt"',
    "cv2": 'uv pip install "inspect-robots-yam[cameras]"',
}
```

The setup wizard checks the configured policy and embodiment when they are
registered, while `doctor` checks its selected embodiment before construction. Both use
`importlib.util.find_spec` without importing the declared top-level modules and
print each remediation command verbatim when a module is missing.

## Declare device slots

Declare device-shaped constructor arguments on the registered embodiment
factory. Import [`DeviceSlot`](/api/#inspect_robots.conformance.DeviceSlot)
from the `inspect_robots.conformance` submodule:

```python
from inspect_robots.conformance import DeviceSlot

DEVICE_SLOTS = (
    DeviceSlot(
        arg="left_channel",
        kind="can",
        label="left arm CAN channel",
        group="arms",
    ),
    DeviceSlot(
        arg="right_channel",
        kind="can",
        label="right arm CAN channel",
        group="arms",
    ),
    DeviceSlot(
        arg="top_cam_device",
        kind="v4l2",
        label="top camera",
        group="cameras",
    ),
    DeviceSlot(
        arg="left_cam_device",
        kind="v4l2",
        label="left camera",
        group="cameras",
    ),
    DeviceSlot(
        arg="right_cam_device",
        kind="v4l2",
        label="right camera",
        group="cameras",
    ),
)
```

The recognized kinds are `v4l2` for stable camera paths, `can` for SocketCAN
interface names, and `serial` for absolute `/dev/serial/by-id` paths. The setup
wizard probes and interviews slots in declaration order, then writes each
selection to its `arg` key under `[embodiment.args]`. Slots with the same
non-`None` `group` are all-or-none. Ungrouped slots remain independent.

## Declare option slots

Declare boolean behavior toggles on the registered embodiment factory. Import
[`OptionSlot`](/api/#inspect_robots.conformance.OptionSlot) from the
`inspect_robots.conformance` submodule:

```python
from inspect_robots.conformance import OptionSlot

OPTION_SLOTS = (
    OptionSlot(
        arg="auto_start",
        label="Skip the operator start prompts (auto_start)",
    ),
)
```

Each slot is one yes/no question in the setup wizard. Its `arg` is the
`[embodiment.args]` key written as `true` or `false`. On re-runs, the carried
config value is the suggested answer. A declaration is skipped when its `arg`
collides with a device slot, a camera key, or an earlier option declaration.

## The conformance kit

Add one test built on
[`assert_embodiment_conformant`](/api/#inspect_robots.conformance.assert_embodiment_conformant)
to your adapter repo:

```python
from inspect_robots.conformance import assert_embodiment_conformant

def test_embodiment_is_conformant() -> None:
    assert_embodiment_conformant(MyEmbodiment().info)
```

Users can audit an installed adapter the same way:

```bash
inspect-robots doctor --embodiment my_arms
```

Both run the same checks, and neither touches hardware (keep your
constructor hardware-free; connect in `reset()`).

### What the kit checks (errors)

| Code | Requirement |
|------|-------------|
| `semantics` | The action `Box` carries `ActionSemantics`. Guardrails and the agent cannot tell absolute targets from displacements without it. |
| `bounds` | Finite `low`/`high` on every dim. Without them the bounds clamp is skipped and no default delta limit can be derived. |
| `dim_labels` | Every dim is named (`("left_j0", ..., "right_gripper")`), uniquely. The agent moves joints by these names. |
| `state_alignment` | Absolute-target modes (`joint_pos`, `eef_abs_pose`) declare exactly one `StateSpec` field with `shape == (action_dim,)`: the proprioceptive reference the agent interpolates from. |
| `guardrails` | `DeltaLimitApprover(action_space)` constructs. This catches absolute pose modes (`eef_abs_pose`) whose rotation representation cannot be clamped per dimension (`quat_*`, `axis_angle`, `euler_xyz`; use `none` or `rot6d`), and displacement pose modes (`eef_delta_pose`) whose rotation representation's identity is not the zero vector (`quat_*`, `rot6d`; use `none`, `axis_angle`, or `euler_xyz`). |

### Warnings

- `control_hz` undeclared: agent motion durations fall back to 10 Hz step
  counting.
- Zero-width dims (`low == high`): nothing can be commanded there.

## What the kit cannot check

Conformance proves your adapter is guardrail-ready and agent-ready. It cannot
prove the declarations are honest. Verify these by hand:

1. **Control mode matches `step()` behavior.** If `step()` adds the action to
   the current position, declare `joint_delta` (or `eef_delta_*`), never
   `joint_pos`. Misdeclaring sends the delta limiter and the agent's motion
   layer down the absolute branch: "hold still" becomes "move by the current
   pose".
2. **Displacement bounds are per-step-sized.** In delta modes the declared
   box is the per-step displacement limit, not the absolute joint limits.
   Reusing absolute limits derives uselessly large swing limits, and an
   asymmetric absolute box (like a `[0, 1]` gripper) clamps one direction of
   motion to zero. Keep an absolute-limit clamp on the summed command inside
   the embodiment as a backstop.
3. **Policy and embodiment declare in lockstep.** If a config flag changes
   the control mode, both sides must build their semantics from the same
   config. A mismatch is a hard compatibility error (good: it fails before
   motion).
4. **Hold behavior between chunks.** Slow policies (VLA servers, LLM agents)
   leave seconds between action chunks. Verify on the rig that motors hold
   position in your configured mode before any unattended run, and keep a
   hand on the e-stop the first time.

## Conventions worth copying

The reference adapters (`inspect-robots-yam`, `plugins/inspect-robots-isaacsim`)
share a shape worth reusing: hardware access behind injected seams (driver
factory, camera reader, clock) so the full suite runs in CI; scalar-only
constructor kwargs so `-E key=value` works from the CLI; a hard safety clamp
inside `step()` independent of any approver; 100% coverage with the real
drivers behind `pragma: no cover` seams; and a preflight or `doctor` run
documented as the first step on new hardware.

---

<!-- Source: guide/cli.md -->

# Command-line interface

The `inspect_robots` CLI wraps the registry and [`eval`](/api/#inspect_robots.eval.eval).
The command is installed as `inspect-robots`, with `inspect-robot` as an alias
for the common typo; both run the same CLI.

## Zero-config: `inspect-robots "<instruction>"`

Once you have configured a default policy and embodiment (run
`inspect-robots setup`, or see below), giving the robot a command is a
one-liner:

```bash
inspect-robots "place the spoon on the plate"
```

This runs a single ad-hoc scene built from that language instruction on
your default policy/embodiment: sugar for
`inspect-robots run --instruction "..."`. The resolved components and where
they came from are printed before the robot moves. Two flags exist only for
instruction runs: `--max-steps N` (horizon, default 300) and `--scorer NAME`
(default `operator`).

The sugar only fires when the first argument contains whitespace, so a
mistyped subcommand (`inspect-robots isnpect`) errors out instead of starting
a rollout; a single-word instruction needs the explicit
`run --instruction "wipe"` form.

### Default policy and embodiment

Resolved in order (first hit wins):

1. explicit flags: `--policy` / `--embodiment`
2. environment: `INSPECT_ROBOTS_POLICY` / `INSPECT_ROBOTS_EMBODIMENT`
3. the user config file `~/.config/inspect-robots/config.ini`
   (`$XDG_CONFIG_HOME` is honored):

```ini
[defaults]
policy = molmoact2
embodiment = yam_arms     ; the default: real hardware
sim_embodiment = my-sim   ; what --sim swaps in (any registered sim embodiment)
scorer = operator         ; optional, ad-hoc runs only
max_steps = 300           ; optional, ad-hoc runs only
store_frames = true       ; optional, capture frames on every run

[policy.args]          ; default -P key=value pairs
server_url = http://gpu-box:8202

[embodiment.args]      ; default -E key=value pairs
top_cam_device = /dev/v4l/by-id/usb-CAM123-video-index0

[sim_embodiment.args]  ; -E pairs used only under --sim
headless = true
```

Values parse like `-P/-E` args (bool/int/float/None/str), `~` expands in
`[*.args]` values, and an explicit `-P/-E key=value` flag overrides the
same-named config key. An `[*.args]` section belongs to the component named
in `[defaults]`: it applies whenever that same component is the one selected
(by default, by flag, or by env var), and is ignored with a stderr note when
a *different* component is selected. Your YAM rig's `rest_pose` never reaches
`--embodiment kitchen`. There is deliberately no project-local config file:
a checked-in file choosing which policy drives your hardware would be a trust
footgun.

### Running in simulation: `--sim`

Real hardware is the default (it is whatever you configured as `embodiment`).
`--sim` swaps in your configured sim counterpart for one invocation:

```bash
inspect-robots "place the spoon on the plate" --sim
inspect-robots run --task my-benchmark --policy molmoact2 --sim
```

The sim embodiment resolves as `$INSPECT_ROBOTS_SIM_EMBODIMENT` > config
`sim_embodiment`, with constructor args from `[sim_embodiment.args]` only:
real-rig args (`[embodiment.args]`: serial ports, camera IDs) never leak into
a sim run, and vice versa. `--sim` together with an explicit `--embodiment`
is an error (they both pick the embodiment); an exported
`$INSPECT_ROBOTS_EMBODIMENT` is simply not consulted under `--sim`: it's a
persistent default for real runs, not a per-invocation intent. The mapping is
explicit configuration: the framework never guesses which sim matches your
robot.

### Operator scoring

An arbitrary instruction has no success oracle, so ad-hoc runs default to the
`operator` scorer. When run on an interactive terminal, the CLI asks after each
trial unless the embodiment already terminated the episode with a definitive
`success` or `failure` verdict. In that case, the CLI records the embodiment's
verdict as the operator judgement instead of asking the operator a second time,
and prints `operator verdict adopted from embodiment: success` (or `failure`) so
the operator can catch a mistaken adoption live.

```text
did the robot succeed? [y/n/partial/skip] (partial scores as failure) n
grader notes (Enter for none): gripper closed early, cube still in frame
```

Prompted verdicts are recorded in the log. The CLI then asks for one optional
line of grader notes. Bare Enter or whitespace-only input records no note. An
adopted embodiment verdict is not followed by a notes prompt, so a self-scoring
embodiment still costs no keypresses per trial.
`skip` records no judgement, but a grader note entered for that trial is still
recorded. Notes never affect the score. Piped/CI stdin or `--no-prompt` never
prompt. A registered `--task` or `eval-set` run prompts only for trials that end
with `termination_reason="operator_end"` — a human pressed the end-episode key — and
never adopts an embodiment verdict or prompts for any other ending, so
unattended runs stay non-blocking. An unjudged trial honestly scores as
failure with "no operator judgement recorded".

## `inspect-robots setup`

The interactive first-run wizard: it prompts for each `[defaults]` key with
a suggested value (Enter accepts, typing overrides), warns when a chosen
policy or embodiment is not registered in the current environment, and then
helps assign camera devices. It lists every color-capable camera that udev
names under `/dev/v4l`, preferring
`/dev/v4l/by-id` names and falling back to port-stable `/dev/v4l/by-path`
names when a by-id link is missing or when two cameras share one serial.
Multi-interface cameras such as the RealSense D435 can lose udev's name race
between their depth and RGB interfaces.

Answer `u` and unplug the camera when asked to identify the physical USB
device that disappeared, including cameras the by-id listing cannot name. The
wizard chooses the stored path after replug because udev reassigns links on
every plug. Answer `p` to switch the listing to port names.

When the selected registered embodiment declares device slots, those slots
drive one device interview for cameras, CAN interfaces, and serial devices.
CAN slots list SocketCAN interfaces and support unplug-to-identify; rigs with
multiple USB adapters named `can0`, `can1`, and so on also receive a udev
pinning suggestion so replug order cannot swap physical devices.

```bash
inspect-robots setup
```

The result is written to `~/.config/inspect-robots/config.ini`
(`$XDG_CONFIG_HOME` honored); an existing file is backed up to
`config.ini.bak` first, and settings the wizard does not manage (such as
`[policy.args]` or `sim_embodiment`) are carried through unchanged. Note
that later `inspect-robots config set` edits drop comments from the file.
The setup command requires an interactive terminal; for scripted
configuration use `inspect-robots config set`.
After writing the config, setup lists missing runtime requirements declared by
the selected registered policy and embodiment, together with their remediation
commands.

Prefer to write the file yourself? This is the wizard's output for a YAM
rig; replace the three camera paths with your rig's V4L2 color nodes
(stable `/dev/v4l/by-id/...` or udev-symlink paths):

```bash
mkdir -p ~/.config/inspect-robots && cat > ~/.config/inspect-robots/config.ini <<'EOF'
[defaults]
policy = molmoact2        # from the inspect-robots-yam plugin
embodiment = yam_arms     # same plugin; cameras configured below
scorer = success_at_end
max_steps = 1200          # 120 s at 10 Hz
rerun = true              # live viewer of cameras/state/actions each run
store_frames = true       # save each run's camera frames under logs/frames/

[embodiment.args]
top_cam_device = /dev/v4l/by-id/YOUR-TOP-CAM
left_cam_device = /dev/v4l/by-id/YOUR-LEFT-CAM
right_cam_device = /dev/v4l/by-id/YOUR-RIGHT-CAM
EOF
```

## `inspect-robots list`

Show registered components (builtins + installed plugins):

```bash
inspect-robots list                 # all kinds
inspect-robots list policies        # just one kind
inspect-robots list embodiments
```

## `inspect-robots run`

Resolve a task/policy/embodiment from the registry and run an eval. Pass
constructor arguments with `-T` (task), `-P` (policy), and `-E` (embodiment) as
`key=value` (parsed as bool/int/float/None/str):

```bash
inspect-robots run --task cubepick-reach --policy scripted --embodiment cubepick
inspect-robots run --task cubepick-reach -T num_scenes=10 --policy scripted -P chunk_size=8 \
             --embodiment cubepick --log-dir logs --seed 0
```

`--epochs N` overrides the task's epoch count, `--fail-on-error X` halts on
`PolicyError`s (`1` = first error, `0<X<1` = proportion, `X>1` = count), and
`--store-frames` streams camera frames to a per-run subdirectory of
`<log-dir>/frames` (trial ids repeat across runs, so each run gets its own
directory; the log's `stats.frames_dir` records the exact path). A
`store_frames = true` config default enables capture on every run;
`--no-store-frames` disables it for one invocation. When the run finishes,
the path of the written log is printed.

`--policy`/`--embodiment` may be omitted when defaults are configured (see
the zero-config section above); `--instruction "..."` replaces `--task` to
run a single ad-hoc scene.

The exit code is `0` on a successful eval, `1` otherwise. When trials errored,
the summary shows the count (`trials: 4 (2 errored)`) and lists each errored
scene; a run in which every trial errored reports `run status: error` and exits `1`.

## `inspect-robots eval-set`

Run several registered tasks against one resolved policy/embodiment pair in a
single invocation — the CLI counterpart of
[`eval_set`](/api/#inspect_robots.eval.eval_set). Task names are matched exactly, or
by shell-quoted `fnmatch` glob (entry-point discovery namespaces tasks as
`<benchmark>/<key>`, so a benchmark name is a ready-made prefix):

```bash
inspect-robots eval-set 'kitchenbench/*' --policy xpolicylab -P url=ws://host:19000 \
             --embodiment yam_arms
inspect-robots eval-set cubepick-reach my-other-task --policy scripted --embodiment cubepick
```

For a task declared with `max_seconds`, its summary row includes both the
physical-time budget and the integer step limit resolved from the selected
embodiment, for example `[120s -> 1200 steps at 10 Hz]`.

Multiple patterns may match the same task; it still runs once. A pattern that
matches nothing is an error listing every registered task. `--policy` and
`--embodiment` (and `-P`/`-E`, `--sim`, `--epochs`, `--fail-on-error`,
`--store-frames`, `--disable-guardrails`, `--max-action-delta`) apply exactly
as they do for `run`, to every matched task — there is no per-task `-T` in
this release. The embodiment is resolved once for the whole set, not once per
task, so a real robot is not reconnected between tasks.

Rather than one full summary per task, the CLI prints the resolved
policy/embodiment, one status line for the whole set, a compact `[status]
task_name  metrics-or-error` row per task, and the shared log directory once
(`eval_set` still writes one `EvalLog` per task inside it). The exit code is
`0` iff every task's log has `status == "success"`.

`--retry-attempts` is accepted and threaded through to `eval_set()`, whose
resumption-of-a-partial-run behavior is reserved for a follow-up: passing it
today does not yet skip already-finished scenes. `--rerun`'s live viewer
is not offered for `eval-set`: streaming several back-to-back tasks into one
viewer window is a separate design question from running the set at all.

## `inspect-robots doctor`

`doctor` reports a registered embodiment's missing declared runtime modules
before constructing it, then checks its spaces for adapter conformance.

```bash
inspect-robots doctor --embodiment my_arms
```

## `inspect-robots inspect`

Print a summary of a saved [`EvalLog`](/api/#inspect_robots.log.EvalLog):

```bash
inspect-robots inspect logs/cubepick-reach_xxxx.json
```

```text
task:        cubepick-reach
policy:      scripted
embodiment:  cubepick
run status:  completed
outcome:     5 succeeded
horizon:     120s -> 1200 steps at 10 Hz
scenes:      5   trials: 5
metrics:
  success_at_end: 1
scenes:
  [success] scene-0: success_at_end=1
  ...
```

The `horizon` line appears for seconds-based tasks; step-only logs retain the
existing output. The HTML viewer likewise separates declared seconds from the
resolved step limit.

`completed` is the display form of the log's `success` status value; the
on-disk field and Python API keep `success`.

For runs whose policy recorded conversations (such as `--policy agent`),
`--transcript` appends each trial's recorded transcript after the summary:

```bash
inspect-robots inspect logs/cubepick-reach_xxxx.json --transcript
```

## `inspect-robots summarize`

Distill a saved [`EvalLog`](/api/#inspect_robots.log.EvalLog) into a markdown
learnings file:

```bash
inspect-robots summarize logs/cubepick-reach_xxxx.json
```

Without `--model`, the command works offline and writes a deterministic digest
of run identity, trial outcomes, operator feedback, errors, and transcript
statistics. The default output is
`logs/learnings/cubepick-reach_xxxx.md`. Use `-o FILE` to select another path
or `-o -` to write only the document to stdout.

With `--model`, the digest and the tail of each recorded policy transcript are
sent to an OpenAI-compatible chat-completions endpoint:

```bash
inspect-robots summarize logs/cubepick-reach_xxxx.json \
  --model claude-sonnet-4-5
```

The default endpoint is `https://api.anthropic.com/v1`, and the default API key
variable is `ANTHROPIC_API_KEY`. Override them with `--base-url URL` and
`--api-key-env VAR` for another compatible provider.

### Retry with learning

The learnings file exists to be fed back in. The
[`agent`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-agent)
and
[`capx`](https://github.com/robocurve/inspect-robots/tree/main/plugins/inspect-robots-capx)
policies accept a `prior_learnings` path and append the file's text to the
system prompt after any embodiment notes, framed as hints that may be stale
(the current observation always wins):

```bash
inspect-robots summarize logs/cubepick-reach_xxxx.json --model claude-sonnet-4-5
inspect-robots "place the fork on the plate" --policy agent \
    -P prior_learnings=logs/learnings/cubepick-reach_xxxx.md
```

The policy reads the file once when it is constructed and records its resolved
path and content hash in the log's policy configuration, so runs that saw
prior notes are never mistaken for cold-start runs when comparing results. Any
hand-written markdown file works in place of a generated one. Validation
details and limits live in the plugin READMEs linked above.

## `inspect-robots view`

Render a saved [`EvalLog`](/api/#inspect_robots.log.EvalLog) as a self-contained HTML
report:

```bash
inspect-robots view logs/cubepick-reach_xxxx.json
```

The report puts the run status, configuration, metrics, scene results, and
recorded policy conversations on one page. Agent notes from tool calls are
highlighted above their call lines. For runs captured with `--store-frames`,
the report also embeds the stored camera frames at the exact observation turns
where the model saw them. The file contains its stylesheet and frame data
inline and uses native browser controls to collapse transcripts, so it has no
network or JavaScript dependency.

By default, `view` replaces the log path's suffix with `.html` and prints the
written path. Use `-o REPORT.html` to choose another file, `-o -` to write only
the HTML document to stdout, or `--open` to launch the written file in the
default browser. Missing output directories are created. The command returns
0 whenever it produces the report, even when the evaluation recorded a failed
or cancelled run.

Frame embedding is on by default when the log's frame directory can be found.
Use `--no-frames` to keep the transcript placeholders, or
`--frames-budget MB` to change the default 50 MB inline-frame payload limit.
`--frames-budget 0` removes the limit. Inlined frames make the HTML document
larger, so use a smaller budget or `--no-frames` when page size matters.

## `inspect-robots video`

Render a `--store-frames` run's stored camera frames into one MP4 per
(trial, camera) stream:

```bash
inspect-robots video logs/adhoc_xxxx.json
```

```text
fps: 10 (control_hz from log)
wrote logs/frames/20260715_184213/scene-0-e0_left_cam.mp4 (1200 frames)
wrote logs/frames/20260715_184213/scene-0-e0_right_cam.mp4 (1200 frames)
wrote 2/2 streams
```

Encoding is done by the `ffmpeg` binary (no Python dependencies are added);
install it from your package manager, or point at a specific build with
`--ffmpeg PATH`. Videos land in the frames directory by default (`--out DIR`
overrides). The playback rate defaults to the log's `control_hz` and can be
overridden with `--fps N`. A stream that fails to encode is reported on
stderr and the remaining streams still encode; the exit code is 1 if any
stream failed.

## `inspect-robots --version`

```bash
inspect-robots --version
```