Skip to main content

API reference

Generated automatically from the source docstrings. The public, stability-guaranteed surface is everything exported by inspect_robots.__all__ (eval, eval_set, read_eval_log, EvalLog and the other log dataclasses); the sections below document the full framework.

Core types & spaces​

inspect_robots.types​

Core observation/action data types exchanged between policy and embodiment.

These are the wire format of a rollout. They are deliberately small, immutable, and NumPy-native. Arrays are raw (the policy owns model-specific preprocessing); images are (H, W, C) uint8.

The dataclasses set eq=False because they carry NumPy arrays, whose element-wise == does not yield a single bool — identity/round-trip semantics are what callers actually need here.

ImageArray​

ImageArray = npt.NDArray[np.uint8]

StateArray​

StateArray = npt.NDArray[np.floating[Any]]

Observation​

class Observation(images: Mapping[str, ImageArray] = dict(), state: Mapping[str, StateArray] = dict(), instruction: str | None = None, image_times: Mapping[str, float] = dict(), state_time: float = 0.0, extra: Mapping[str, Any] = dict()):

A single multi-modal observation produced by an embodiment.

images are keyed by camera name; state holds proprioception keyed by a controlled vocabulary (e.g. "eef_pos", "gripper"). instruction is the language goal for this step (usually constant across an episode, but may change for long-horizon tasks). The rollout injects extra["env_step"] (int), extra["approvals"] (list of {"t": int, "detail": str | None} records for safety interventions since the previous act() call), and extra["operator_messages"] (list of {"t": int, "text": str, "source": str} records for live feedback since the previous act() call) into the policy-facing observation and reserves those keys; embodiments should not set them.

Action​

class Action(data: StateArray, meta: Mapping[str, Any] = dict()):

A single action to apply to an embodiment.

Semantics (control mode, rotation representation, gripper kind, frame) live on the action space, not on every action instance — see inspect_robots.spaces.

ActionChunk​

class ActionChunk(actions: Sequence[Action], control_hz: float | None = None, inference_latency_s: float | None = None, meta: Mapping[str, Any] = dict()):

A horizon of actions predicted by one policy inference.

Modern VLAs (π0, ACT, diffusion policies) predict H future actions that are executed open-loop because inference is slower than the control rate. H == 1 is the degenerate "reactive policy" case. control_hz is the rate the chunk was intended to be played at — advisory metadata a consumer may honor (the rollout enforces no rate; None leaves it unspecified). inference_latency_s, when measured, is logged.

OPERATOR_END​

OPERATOR_END = 'operator_end'

StepResult​

class StepResult(observation: Observation, reward: float | None = None, terminated: bool = False, termination_reason: str | None = None, truncated: bool = False, info: Mapping[str, Any] = dict()):

The outcome of applying one action to an embodiment.

terminated means the task ended (success or hard failure); termination_reason disambiguates (e.g. "success", "collision", "fault", "out_of_bounds", "operator_end" — the standard reason for operator-ended episodes awaiting a verdict). truncated means a time/horizon cutoff. A simulator may expose privileged success via info.

inspect_robots.spaces​

Action/observation spaces and action semantics.

Spaces describe the shape of actions and observations; ActionSemantics describes what an action means (control mode, rotation representation, gripper kind, reference frame). Semantics are what make compatibility checking real (a 7-DoF VLA vs a 6-DoF arm; delta vs absolute poses) and make temporal ensembling correct.

This module ships a minimal-but-functional core for the tracer slice; richer validation and the full StateSpec vocabulary are layered on in a later step without changing these signatures.

ControlMode​

ControlMode = Literal['joint_pos', 'joint_vel', 'joint_delta', 'eef_delta_pose', 'eef_abs_pose', 'eef_delta_pos']

RotationRepr​

RotationRepr = Literal['none', 'quat_wxyz', 'quat_xyzw', 'rot6d', 'axis_angle', 'euler_xyz']

GripperKind​

GripperKind = Literal['none', 'continuous', 'binary']

Frame​

Frame = Literal['base', 'world', 'camera']

ABSOLUTE_CONTROL_MODES​

ABSOLUTE_CONTROL_MODES = frozenset({'joint_pos', 'eef_abs_pose'})

ActionSemantics​

class ActionSemantics(control_mode: ControlMode, rotation_repr: RotationRepr = 'none', gripper: GripperKind = 'none', frame: Frame = 'base', dim_labels: tuple[str, ...] | None = None, max_step: tuple[float | None, ...] | None = None):

What an action vector means. Attached to an action Box.

dim_labels optionally names each action dimension (e.g. ("left_j0", ..., "right_gripper") for a bimanual arm) so tooling — LLM agent policies, logging, visualization — can address dimensions by name. Length is validated against the owning box in Box.__post_init__ (the semantics/shape pairing is only visible there).

max_step optionally declares the safe per-control-step change for each absolute-target dimension in the action space's native units. A None entry leaves that dimension to range-based defaults. Displacement and rate boxes already declare per-step limits through their bounds, so declarations are rejected for those modes.

Box​

class Box(shape: tuple[int, ...], low: npt.NDArray[np.floating[Any]] | None = None, high: npt.NDArray[np.floating[Any]] | None = None, semantics: ActionSemantics | None = None):

A continuous box-shaped space. Optional low/high bounds and, for action spaces, ActionSemantics.

CameraSpec​

class CameraSpec(name: str, height: int, width: int, channels: int = 3):

An image stream an embodiment provides or a policy requires.

CANONICAL_STATE_UNITS​

CANONICAL_STATE_UNITS: dict[str, str] = {'joint_pos': 'rad', 'joint_vel': 'rad/s', 'eef_pos': 'm', 'eef_pose': 'm+quat', 'eef_quat': 'unit_quat', 'gripper': 'normalized', 'gripper_width': 'm'}

StateField​

class StateField(key: str, shape: tuple[int, ...], unit: str = '', dtype: str = 'float64'):

One proprioception field: its key, shape, unit, and dtype.

StateSpec​

class StateSpec(fields: tuple[StateField, ...] = ()):

A richer description of an embodiment's proprioception than a bare key set.

ObservationSpace​

class ObservationSpace(cameras: tuple[CameraSpec, ...] = (), state_keys: frozenset[str] = frozenset(), state: StateSpec | None = None):

The observations an embodiment provides / a policy requires.

state_keys is the compatibility-relevant set of proprioception keys. state optionally carries the richer StateSpec (shapes/units).

Policy & embodiment​

inspect_robots.policy​

The Policy (VLA) interface — one of Inspect Robots's two swappable inputs.

A Policy is the "brain": given an Observation (plus the scene's instruction), it returns an ActionChunk to be executed open-loop.

The public contract is a runtime-checkable Policy Protocol so callers can wrap existing models without inheriting. PolicyBase is an optional convenience ABC with sane defaults.

PolicyConfig​

class PolicyConfig(action_horizon: int = 1, replan_interval: int | None = None, temperature: float | None = None):

Inference-time configuration, recorded in the eval log.

The VLA analog of Inspect's GenerateConfig: action-chunk handling and sampling knobs that affect reproducibility.

PolicyInfo​

class PolicyInfo(name: str, action_space: Box, observation_space: ObservationSpace = ObservationSpace(), control_hz: float | None = None):

Static description of a policy used for compatibility checking + logging.

Policy​

class Policy(Protocol):

The VLA contract.

Policies may additionally define five optional hooks, none part of this Protocol so existing policies stay conformant. bind(embodiment_info) lets embodiment-adaptive policies adopt the embodiment's spaces; eval() calls it after resolving both components and before compatibility checking. on_trial_start(scene_id, epoch, log_dir, run_id) runs immediately before each trial's rollout. It is a policy lifecycle hook, distinct from the sink bus hook of the same name, which runs first and takes only scene id and epoch. on_trial_end(record, log_dir, run_id) runs when a trial finishes (including errored and cancelled trials, except trials whose on_trial_start raised, which never reached reset()), before sinks see the record. Mutations to record.metadata land in the log, and exceptions degrade the run to status="error" rather than crashing the overall evaluation. transcript() returns a small JSON-serializable audit record for the current trial, such as an LLM conversation. The framework calls it once per trial at trial end after a successful reset(), including errored trials. It must be idempotent and safe between resets, must not mutate policy state, and its return value must not alias live state. Camera images must not be embedded because frame sidecars already persist them. Collection runs on the rollout thread and is best-effort: the framework normalizes and bounds the result, and a raising or misbehaving hook cannot change trial outcome. transcript_delta() returns plain-JSON-type messages appended since its previous call, or since reset() on the first call, and returns None or an empty list when nothing is new. Implementations must sanitize only the new slice in O(new messages), including eliding image bytes before the result reaches visualization sinks, and reset() must rewind the cursor. PolicyBase ships defaults for bind() and transcript() but deliberately has no transcript_delta() default: policies must opt in so every inference does not pay for a no-op hook call.

PolicyBase​

class PolicyBase(ABC):

Optional base class providing defaults; inherit only for the helpers.

inspect_robots.embodiment​

The Embodiment interface — Inspect Robots's second swappable input.

An Embodiment is the "body + world": a real robot or a simulator. It produces observations, executes actions, and owns the action/observation spaces, the native control rate, and reset/safety machinery.

Designed around real-robot reality: reset may drive to a home pose and block on human confirmation; there is no guaranteed privileged success oracle. Simulators are a stricter special case that opt into extra capabilities.

Per R1 (see the design doc): rollout() applies no wall-clock pacing of its own — step() is called as fast as the policy/controller/approver can produce actions. An embodiment that needs real-time cadence (e.g. matching a real robot's control period) paces itself inside step(), and declares the "self_paced" capability to document that it does.

Capability​

Capability = str

SEEDABLE​

SEEDABLE: Capability = 'seedable'

RESETTABLE​

RESETTABLE: Capability = 'resettable'

AUTO_RESET​

AUTO_RESET: Capability = 'auto_reset'

PRIVILEGED_SUCCESS​

PRIVILEGED_SUCCESS: Capability = 'privileged_success'

RENDERABLE​

RENDERABLE: Capability = 'renderable'

SELF_PACED​

SELF_PACED: Capability = 'self_paced'

EmbodimentInfo​

class EmbodimentInfo(name: str, action_space: Box, observation_space: ObservationSpace, control_hz: float | None = None, is_simulated: bool = False, capabilities: frozenset[Capability] = frozenset(), supported_setups: frozenset[str] = frozenset(), supported_target_kinds: frozenset[str] = frozenset(), docs: str | None = None):

Static description of an embodiment for compatibility checking + logging.

docs contains free-form markdown operating notes for policies that can read text, such as joint layout and positive directions, zero-pose geometry, gripper polarity, frame conventions, and workspace hints. Keep it concise because consumers inject it into system prompts verbatim. None means no notes are offered; consumers must treat empty and whitespace-only strings the same as absence.

Embodiment​

class Embodiment(Protocol):

The robot/simulator contract.

Embodiments may additionally define an optional bind_task(envelope) hook (not part of this Protocol, so existing embodiments stay conformant): after compatibility checking, eval() calls it with the task's resolved TaskEnvelope before the first reset, letting adapters learn the rollout horizon — e.g. an operator countdown showing elapsed/total. A seconds-based task is converted to integer steps using info.control_hz before the hook fires. The hook is optional input, not a guarantee: it never fires on direct rollout() calls or on older cores, so adapters must keep a graceful fallback. It fires once per eval(), which can be several times over an embodiment's lifetime; each call replaces the previous envelope. EmbodimentBase ships a no-op default.

Embodiments may additionally offer an optional connect_operator_session(session) hook. Accepting the session is a stand-down promise for that run: the embodiment must never read stdin or print its own status output after the call. The session provides status(line) for an in-place status line, write_line(text) for scrollback that safely repaints that status, and gate(prompt, hint=...) for blocking readiness confirmation. A gate flushes stale fd input first and raises EmbodimentFault when stdin is unavailable, with remedies and the embodiment's optional hint. The hook is optional input, not a guarantee. It never fires under --no-prompt, without a TTY, on Windows, from direct rollout() or eval() calls, or on older cores. An embodiment must keep a graceful fallback and a working end-of-episode path whenever the hook does not fire. The hook stays outside this Protocol so older embodiments remain conformant.

Embodiments may additionally offer an optional observe_parked() hook: when a scored trial is about to be graded, eval() calls it so the embodiment can move itself to its parked/rest pose and return one fresh Observation whose cameras see the scene unobstructed; the vlm grader then judges those final frames instead of the last step's. Returning None declines (right when parking would alter the state being graded, e.g. a held object). Raising SafetyAbort or EmbodimentFault halts the eval; any other failure degrades to last-step frames with a RuntimeWarning. The hook is optional input, not a guarantee: it never fires on direct rollout() calls, without a grading hook, or on older cores, and it stays outside this Protocol so older embodiments remain conformant.

The optional defer_operator_end() hook is superseded by connect_operator_session but remains supported. An embodiment that polls stdin for the end-of-episode keypress must stop doing so when called; the framework console owns stdin for that run. Offering this legacy hook is also how a real-hardware embodiment declares itself console-safe to cores and adapters that have not adopted the session hook.

Embodiments may also define an optional contribute_guardrails(self, action_space: Box) -> GuardrailContribution hook, likewise omitted from this Protocol so existing implementations remain runtime-conformant. The CLI calls it with the resolved action space and appends its contribution after the generic clamp and delta limiter. Direct rollout() and programmatic eval() callers compose their own approvers, so the hook is CLI input rather than a universal callback. EmbodimentBase returns an empty contribution by default.

EmbodimentBase​

class EmbodimentBase(ABC):

Optional base class with a no-op close; inherit for the convenience.

User configuration​

inspect_robots.defaults​

Read the user configuration shared by Inspect Robots and plugin CLIs.

Component names from environment variables override names from <config-home>/inspect-robots/config.ini, but they never override the args owners recorded by that file. An args section is valid only for its owner — apply embodiment_args only after confirming the owner is the embodiment you drive, or the args recorded for someone else's rig will configure yours:

defaults = load_defaults(os.environ)
if defaults.embodiment_args_owner == "my_embodiment":
    args = defaults.embodiment_args

(Plugins with several embodiments can resolve the owner through inspect_robots.registry.registered and check its class instead of comparing names.)

INSPECT_ROBOTS_CONFIG selects the config file itself before the standard config-home derivation.

One exception to the owner rule: taskgen_args ([taskgen.args]) has no owner. Automatic task generation is a single fixed function, not a registry-selected component, so there is no differently-selected component for its args to leak into; the section applies to every --auto-task run.

A missing file yields empty defaults. A malformed or type-invalid file raises SystemExit naming the file, with a plain one-line message that callers may catch.

Defaults​

class Defaults(policy: str | None = None, policy_source: str | None = None, embodiment: str | None = None, embodiment_source: str | None = None, sim_embodiment: str | None = None, sim_embodiment_source: str | None = None, scorer: str | None = None, grader: str | None = None, max_steps: int | None = None, store_frames: bool = False, rerun: bool = False, rerun_save: bool = True, rerun_port: int | None = None, policy_args: dict[str, Any] = dict(), embodiment_args: dict[str, Any] = dict(), sim_embodiment_args: dict[str, Any] = dict(), grader_args: dict[str, Any] = dict(), policy_args_owner: str | None = None, embodiment_args_owner: str | None = None, sim_embodiment_args_owner: str | None = None, grader_args_owner: str | None = None, taskgen_args: dict[str, Any] = dict()):

Resolved user defaults, each with a human-readable source for the run header.

config_path​

def config_path(env: Mapping[str, str]) -> Path | None:

Return the user config file path derived from env.

INSPECT_ROBOTS_CONFIG names the config file itself and takes precedence over XDG_CONFIG_HOME and HOME. Otherwise, the result is <config-home>/inspect-robots/config.ini, whether or not the file exists. Return None when none of those variables is set; a variable set to the empty string counts as unset.

load_defaults​

def load_defaults(env: Mapping[str, str]) -> Defaults:

Load user defaults: environment variables override the config file.

env is injected (pass os.environ) so tests never touch the real home directory. Environment variables override component names without changing the config file's args owners. A missing config file yields empty defaults; a malformed or type-invalid one raises SystemExit naming the file. The exception carries a plain one-line message and is catchable by diagnostic callers that need to continue other checks.

Tasks & scenes​

inspect_robots.scene​

Scenes — the robotics analog of Inspect AI's Sample.

A Scene is one initial condition of a benchmark: a language instruction, an optional success Target, an optional seed, and metadata. A benchmark Task iterates over a dataset of scenes (e.g. 50 object layouts), repeated epochs times.

Field mapping to Inspect: Sample(input, target, id, metadata, setup) ↔ Scene(instruction, target, id, metadata, setup, init_seed).

Target​

class Target(kind: str, spec: Mapping[str, Any] = dict()):

A success specification the scorer reads. Embodiment-namespaced.

kind names what the embodiment must realize/evaluate (e.g. "reach_object"); spec carries the parameters. Kept intentionally open for the tracer; richer typed targets land with the scorer milestone.

Scene​

class Scene(id: str, instruction: str, target: Target | None = None, init_seed: int | None = None, setup: str | None = None, metadata: Mapping[str, Any] = dict()):

One initial condition of a benchmark.

ListSceneDataset​

class ListSceneDataset(scenes: Sequence[Scene]):

A trivial in-memory scene dataset backed by a sequence.

inspect_robots.task​

The Task — an embodiment-agnostic benchmark definition.

Mirrors Inspect AI's Task = dataset + scorer + epochs/reducer, adapted for robotics: the dataset is a sequence of Scene initial conditions and the rollout horizon (max_steps) lives here.

Epochs​

class Epochs(count: int = 1, reducer: str = 'mean'):

Repeat count plus the reducer used to combine per-epoch scores.

Mirrors Inspect's Epochs(count, reducer); reducer is a registered name (default "mean").

TaskEnvelope​

class TaskEnvelope(name: str, max_steps: int):

Identity and rollout limits of a task, safe to hand to adapters.

This is what eval() passes to an embodiment's optional bind_task(envelope) hook: enough for the adapter to display or pre-allocate for the run (e.g. an operator countdown against max_steps), and nothing that would let it second-guess scoring or the dataset. Deliberately carries no control rate — the rollout enforces no wall-clock rate of its own (R1, revised); a self-paced embodiment owns its own cadence.

Task​

class Task(name: str, scenes: Sequence[Scene], scorer: Scorer | str | Sequence[Scorer | str], max_steps: int | None = None, epochs: int | Epochs = 1, metadata: Mapping[str, Any] = dict(), *, max_seconds: float | None = None):

A benchmark: scenes + scorer(s) + horizon, independent of any embodiment.

scorer accepts scorer objects or registry names (e.g. scorer="success_at_end"), or a sequence mixing both.

Declare exactly one rollout horizon. max_steps is already resolved; max_seconds is keyword-only and is resolved against the paired embodiment's positive finite control_hz by eval().

inspect_robots.taskgen​

Generate a scene whose instruction goes to the policy and rubric goes to the grader.

The returned Scene is a two-consumer contract: scene.instruction tells the policy what to attempt, while scene.metadata["rubric"] tells a grader how to judge the final state.

generate_scene​

def generate_scene(embodiment: Embodiment, *, model: str | None = None, instructions: str | None = None, instructions_file: str | None = None, base_url: str = 'https://api.anthropic.com/v1', api_key_env: str = 'ANTHROPIC_API_KEY', max_cameras: int = 4, scene_id: str = 'auto-0', seed: int | None = 0, http_post: HttpPost | None = None, effort: str | float | None | _Unset = _UNSET) -> Scene:

Peek at one seeded world and return a policy instruction plus grader rubric.

seed is the eval seed the caller will pass to eval. The peek uses its first-trial derived seed so a seedable embodiment presents the same initial world to generation and evaluation. effort rides the request as reasoning_effort: leaving it unset omits the field for the provider default, None (-A effort=none) requests the minimum, any other value passes through verbatim, and "" is a guided error. Configuration, transport, and reply-contract failures raise ConfigError before rollout starts.

Scoring​

inspect_robots.scorer​

Scoring: Scores, the Scorer protocol, epoch reducers, and builtin scorers.

Mirrors Inspect AI's @scorer/reducer split. A scorer maps a recorded trajectory (+ the scene's Target) to a Score; an epoch reducer collapses the per-epoch scores of one scene into a single score before metrics aggregate across scenes.

Scorers consume the recorded trajectory (not a live environment), so scoring is reproducible from a saved log.

ScoreValue​

ScoreValue = bool | int | float | str

Reducer​

Reducer = Callable[[Sequence['Score']], 'Score']

Score​

class Score(value: ScoreValue, explanation: str | None = None, metadata: Mapping[str, Any] = dict()):

The outcome a scorer assigns to one trajectory.

value_to_float​

def value_to_float(value: ScoreValue) -> float:

Coerce a score value to a float for metric aggregation.

Scorer​

class Scorer(Protocol):

Maps a recorded trajectory + scene target to a Score.

reduce_mean​

def reduce_mean(scores: Sequence[Score]) -> Score:

Collapse numeric epoch values with the arithmetic mean.

reduce_median​

def reduce_median(scores: Sequence[Score]) -> Score:

Collapse numeric epoch values with the median.

reduce_max​

def reduce_max(scores: Sequence[Score]) -> Score:

Keep the largest numeric epoch value.

reduce_min​

def reduce_min(scores: Sequence[Score]) -> Score:

Keep the smallest numeric epoch value.

reduce_mode​

def reduce_mode(scores: Sequence[Score]) -> Score:

Most common raw value (works for categorical scores). Deterministic.

pass_at_k​

def pass_at_k(k: int) -> Reducer:

Unbiased pass@k estimator over the epoch scores (success = value >= 0.5).

get_reducer​

def get_reducer(name: str) -> Reducer:

Resolve a builtin reducer or parse a dynamic pass_at_<k> name.

reduce_scores​

def reduce_scores(name: str, scores: Sequence[Score]) -> Score:

Apply the named epoch reducer to one scene's scores.

success_at_end​

def success_at_end() -> Scorer:

Score 1.0 iff the episode terminated with reason "success".

episode_length​

def episode_length() -> Scorer:

Score = number of environment steps taken.

min_distance_to_goal​

def min_distance_to_goal() -> Scorer:

Score = the closest the effector got to the goal (lower is better).

reached_goal_state​

def reached_goal_state(threshold: float = 0.05) -> Scorer:

Success iff the effector came within threshold of the goal.

is_affirmative_verdict​

def is_affirmative_verdict(verdict: str | None) -> bool:

Whether a recorded operator judgement reads as "the trial succeeded".

The single public definition of that contract: the recognized affirmative vocabulary and the comparison rules around it (surrounding whitespace is ignored, matching is case-insensitive, and None — no judgement recorded — is not affirmative). Benchmarks that grade real-world runs from operator verdicts should call this rather than restate any part of it, so a change here reaches every consumer at once.

operator_scorer​

def operator_scorer() -> Scorer:

Score from the human operator's recorded success judgement (R6).

VLMScorer​

class VLMScorer:

Reserved interface (R10): score from a VLM classifier over final frames.

VLM judging shipped as a grader instead (R6: scorers must stay pure readers of the record); instantiating and calling this raises so the reserved contract stays visible without half-baked behavior.

inspect_robots.grader​

Judgement capture as a component: the capture side of R6 (plan 0049).

R6 splits judging in two. A grader captures the success judgement once per scored trial, after the rollout returns and before scorers run; it may be interactive (a human prompt) or expensive (a VLM call), and it mutates the TrialRecord. Scorers stay pure readers of the record, so re-scoring a saved log remains deterministic.

Grader​

class Grader(Protocol):

Capture a judgement onto a trial record, once, before scorers run.

grade is called only for trials that will be scored (never for errored or cancelled trials) and may mutate the record: set operator_judgement/operator_note and append an operator event. It must tolerate being unable to grade (e.g. no judge available) by leaving the record unchanged rather than raising.

operator_grader​

def operator_grader(session: OperatorSession | None = None) -> _OperatorGrader:

The builtin human grader: prompt for a verdict and optional notes.

Adopts a console- or embodiment-definitive verdict without re-asking, and never blocks a run whose stdin cannot answer (EOF degrades to no verdict).

vlm_grader​

def vlm_grader(model: str, rubric: str | None = None, rubric_file: str | None = None, *, base_url: str = 'https://api.anthropic.com/v1', api_key_env: str = 'ANTHROPIC_API_KEY', max_cameras: int = 4, http_post: HttpPost | None = None, effort: str | float | None | _Unset = _UNSET) -> _VLMGrader:

The builtin automated grader: a vision model judges first and last frames.

Configuration fails fast here (missing model, unreadable rubric file, unset API key, empty effort) so a misconfigured run stops before any rollout; after the rollout its grade only ever degrades to an ungraded trial, so an effort value the endpoint rejects surfaces as a per-trial stderr note, not a raise. effort rides each grading request as reasoning_effort: leaving it unset omits the field for the provider default, None (-G effort=none) requests the minimum, and any other value passes through verbatim. The judgement lands on the same record fields the operator grader writes, so the operator scorer reads it unchanged.

Rollout, controllers & safety​

inspect_robots.rollout​

The rollout engine — the closed control loop at the heart of Inspect Robots.

One rollout runs a single trial (one scene, one epoch): it drives the policy↔embodiment loop through the Controller (open-loop chunk execution) and the Approver safety gate, logging each step to the sinks, and returns an immutable TrialRecord that scorers consume.

derive_seed​

def derive_seed(eval_seed: int | None, scene_seed: int | None, epoch: int) -> int:

Deterministically combine eval/scene seeds and the epoch index (R2).

Distinct epochs of the same scene get distinct seeds so repeats actually vary for stochastic policies, while a fixed (eval_seed, scene_seed, epoch) reproduces bitwise. None and 0 hash differently, so an unseeded input does not silently alias seed=0.

StepRecord​

class StepRecord(t: int, observation: Observation, action: Action, result: StepResult, image_refs: Mapping[str, FrameRef] | None = None, result_image_refs: Mapping[str, FrameRef] | None = None):

One step of a recorded trajectory.

When a FrameStore is used, both observation and result.observation have their images stripped. image_refs and result_image_refs hold the corresponding on-disk handles instead (R5).

TrialRecord​

class TrialRecord(scene_id: str, epoch: int, seed: int | None, steps: list[StepRecord] = list(), parked_observation: Observation | None = None, terminated: bool = False, truncated: bool = False, termination_reason: str | None = None, status: str = 'success', error: str | None = None, inference_latencies: list[float] = list(), operator_judgement: str | None = None, operator_note: str | None = None, events: list[Event] = list(), metadata: dict[str, Any] = dict(), policy_transcript: Any = None):

The full record of one trial — the unit scorers consume.

rollout​

def rollout(policy: Policy, embodiment: Embodiment, scene: Scene, *, max_steps: int, seed: int | None, epoch: int, controller: Controller, approver: Approver, sink: LogSink, frame_store: FrameStore | None = None, operator_input: OperatorInput | None = None) -> TrialRecord:

Run a single trial and return its record.

Generic exceptions raised by the policy, or a non-finite action, are reported as PolicyError; generic exceptions raised by the embodiment are reported as EmbodimentFault; by the approver as SafetyAbort (an approver that crashed cannot vouch for safety), as is a non-finite action introduced by the approver. Already-typed Inspect Robots errors (incl. SafetyAbort) propagate unchanged, so the eval orchestrator can apply the correct continue-vs-halt policy. Every error raised from inside the trial carries the partial TrialRecord on exc.record for the orchestrator to preserve.

The loop applies no wall-clock pacing of its own: embodiment.step() is called as fast as the policy/controller/approver can produce actions. An embodiment that needs real-time cadence paces itself inside step() and declares the "self_paced" capability to document that it does (see Embodiment).

inspect_robots.controller​

Controllers — the rollout middleware layer (Inspect's @solver analog).

A Controller owns the per-control-step decision of which action to send to the embodiment. It internally decides when to call policy.act() (a slow VLA inference returning an ActionChunk), buffers the returned chunk, and pops the next action each step. This single-method, stateful shape (R3) is what lets advanced controllers — e.g. a temporal-ensembling controller that re-infers every step and blends overlapping predictions — compose without forking the rollout loop.

DefaultController plays the first replan_interval actions of each chunk, then re-infers (replan_interval=None ⇒ play the whole chunk before replanning).

Controller​

class Controller(Protocol):

Decides the next action to execute, calling the policy as needed.

DefaultController​

class DefaultController(replan_interval: int | None = None):

Open-loop chunk execution with periodic replanning.

SmoothingController​

class SmoothingController(inner: Controller, alpha: float = 0.5):

Wrap another controller and exponentially smooth its action stream.

Demonstrates the middleware composition the single-method interface enables: the wrapped controller owns inference/replanning while this layer applies an exponential moving average (alpha toward the new action) on top. Only valid for additive/continuous action spaces (the caller's responsibility).

EnsemblingController​

class EnsemblingController(action_space: Box, m: float = 0.1):

ACT/ALOHA-style temporal ensembling over overlapping action chunks.

Queries the policy every control step and blends, for the current step, the predictions of all still-relevant recent chunks. A chunk queried at global step q predicts step t via its action at index t - q (valid while 0 <= t - q < len(chunk)). Predictions are weighted exp(-m * i) with i = 0 for the oldest contributing chunk (ALOHA's convention: older predictions dominate, which smooths motion); larger m favors the oldest.

Only valid for additive action representations: the constructor refuses rotation reps and binary grippers that cannot be linearly averaged (R8).

inspect_robots.approver​

The Approver — a safety gate between policy output and the embodiment.

Every action passes through Approver.review before embodiment.step. This is the robotics analog of Inspect AI's ApprovalPolicy and is more safety-critical: an approver may pass, clamp, or veto an action (a veto raises SafetyAbort). In the tracer slice the default approver passes everything through; clamping/operator approval land in rollout hardening.

Approver​

class Approver(Protocol):

Reviews an action before it reaches the embodiment.

May return the action unchanged, return a modified (e.g. clamped) action, or raise SafetyAbort to halt the eval.

GuardrailContribution​

class GuardrailContribution(approvers: tuple[tuple[str, Approver], ...] = (), warnings: tuple[str, ...] = ()):

Approvers an embodiment adds to the CLI's default guardrail chain.

approvers pairs a short display name (shown in the guardrails: banner) with each approver to append. warnings names contributions the embodiment declined to make or made in a degraded state, and why, so neither condition is invisible. Warnings may accompany active approvers.

A contributed approver must preserve the incoming Action object's identity when approving it unmodified. If it substitutes another target, such as holding an earlier pose, it must call DeltaLimitApprover.rewind_reference with that executed pose so later deltas are measured from reality. It vetoes by raising SafetyAbort; other exceptions are bugs and propagate. The rollout enforces finiteness before review and again on the returned action, so contributed approvers need not duplicate that validation.

AutoApprover​

class AutoApprover:

Approve every action unchanged (the permissive default).

ClampApprover​

class ClampApprover(action_space: Box):

Clamp actions to a box's low/high bounds before they reach hardware.

One-sided boxes are honored: a low-only box clamps from below, a high-only box from above. A modified action is flagged via action.meta["clamped"] so the rollout can record an approval event; when nothing clamps, the same action object is returned (the rollout detects modification by identity).

Non-finite values are the safety cases for direct review() callers, since rollout rejects non-finite policy actions before review: NaN anywhere in the action raises SafetyAbort. A NaN is a poisonous value with no meaningful clamp, and it must never reach hardware. ±inf is not an abort: it clamps to the finite bound on that side like any other out-of-range value (and passes through if that side is unbounded).

DeltaLimitApprover​

class DeltaLimitApprover(action_space: Box, max_delta: float | Any | None = None):

The "no wild swings" gate — semantics-aware per-step change limiting.

Absolute-target modes (joint_pos, eef_abs_pose) clamp each dimension to at most max_delta away from the last approved action of the trial; the first action passes through un-delta-limited (there is no trustworthy reference yet — bounds clamping still applies upstream). The derived default is a declared ActionSemantics.max_step where present, otherwise 5% of high - low per step.

Displacement/rate modes (eef_delta_pos, eef_delta_pose, joint_delta, joint_vel) are the per-step change, so each dimension clamps to the intersection of the box and [-max_delta, +max_delta]. The derived default is the box alone — core cannot assume displacement bounds are per-step-sized, so without an explicit max_delta the limiter adds nothing beyond ClampApprover.

Construction never guesses: missing semantics, a needed missing/non-finite bound without an explicit max_delta, an absolute pose mode whose rotation representation cannot be clamped per-dimension, or a displacement pose mode carrying a quaternion or rot6d delta (whose identity is not the zero vector, so per-dimension clamping distorts it) all raise ValueError. NaN anywhere in a reviewed action raises SafetyAbort. A modified action is flagged meta["delta_clamped"]; an unmodified one is returned as the same object (rollout detects modification by identity). The reference lives in the rollout store (fresh per trial) under a namespaced key.

ChainApprover​

class ChainApprover(*approvers: Approver):

Run approvers in sequence, feeding each the previous one's result.

Gives "guardrails" one composable name, e.g. ChainApprover(ClampApprover(space), DeltaLimitApprover(space)). Identity is preserved end to end: if no approver modifies the action, the original object comes back, so the rollout's modification check still works.

inspect_robots.frames​

FrameStore — rollout-owned streaming of camera frames to disk (R5).

A long multi-camera episode would exhaust memory if every frame were retained in the TrialRecord. Instead the rollout streams frames to disk through a FrameStore and keeps only lightweight FrameRef handles. This is owned by the rollout, NOT by any log sink, so trajectories are recorded (and scorable) independent of which optional sinks are enabled.

FrameRef​

class FrameRef(camera: str, t: int, path: str):

A handle to a camera frame stored on disk.

FrameStore​

class FrameStore(root: str):

Persist frames as .npy files under root and hand back refs.

inspect_robots.transcript​

A typed transcript of rollout events.

Each trial records an ordered stream of events (reset, inference, step, approval, operator feedback, operator judgement, error). This is the robotics analog of Inspect AI's transcript and is the data a results viewer renders. Events are deliberately lightweight: a kind, the step index t (-1 for pre-loop events), and a small data payload.

EventKind​

EventKind = str

Event​

class Event(kind: EventKind, t: int, data: Mapping[str, Any] = dict()):

One entry in a trial's transcript.

reset_event​

def reset_event(seed: int | None) -> Event:

Record the seed used before control step zero.

inference_event​

def inference_event(t: int, latency_s: float | None, chunk_len: int) -> Event:

Record one policy call's timing and buffered action count at step t.

step_event​

def step_event(t: int, terminated: bool, truncated: bool, reason: str | None) -> Event:

Record termination state after executing control step t.

approval_event​

def approval_event(t: int, modified: bool, detail: str | None = None) -> Event:

Record whether the safety gate changed the proposed action at step t.

operator_message_event​

def operator_message_event(t: int, text: str, source: str = 'console') -> Event:

Record live feedback typed at the console or spoken in voice mode.

This is distinct from the post-hoc operator verdict event.

operator_event​

def operator_event(t: int, verdict: str, source: str = 'prompt', note: str | None = None) -> Event:

Record the operator's verdict, optional note, and source after the rollout ends.

verdict carries whatever the operator answered, which includes the non-verdict "skip" when they declined to grade a trial but still left a note. Consumers deriving an outcome from this event must treat "skip" as "no judgement", never as a result.

judgement_source​

def judgement_source(record: TrialRecord) -> str | None:

Return which path produced the record's judgement.

None when the record has no operator_judgement, or when the judgement was set without an operator event; otherwise the source of the last operator event ("console", "prompt", "embodiment", "vlm").

error_event​

def error_event(t: int, error_type: str, message: str) -> Event:

Record an exception's type and message at the failing control step.

Compatibility & errors​

inspect_robots.compat​

Compatibility checking between a policy and an embodiment.

Before any rollout, Inspect Robots verifies that a (policy, embodiment) pair can actually run together: the action spaces agree in dimension and semantics, the embodiment provides every observation the policy requires (resolving a name remap), the control rates are reconcilable (R1), and — given a task — every scene is realizable on the embodiment (R7).

Hard mismatches are error issues that fail fast; soft ones are warnings.

CompatIssue​

class CompatIssue(severity: str, code: str, message: str):

One compatibility finding with a stable code and human-readable detail.

CompatibilityReport​

class CompatibilityReport(issues: list[CompatIssue] = list(), remap: dict[str, str] = dict()):

The outcome of a compatibility check.

check_compatibility​

def check_compatibility(policy: Policy, embodiment: Embodiment, task: Task | None = None, *, remap: dict[str, str] | None = None) -> CompatibilityReport:

Return a structured compatibility report (does not raise).

assert_compatible​

def assert_compatible(policy: Policy, embodiment: Embodiment, task: Task | None = None, *, remap: dict[str, str] | None = None) -> CompatibilityReport:

Check compatibility and raise CompatibilityError on hard errors.

inspect_robots.errors​

Inspect Robots error taxonomy.

The split below resolves the "fail fast vs never-crash-overnight" tension:

  • ConfigError / CompatibilityError are raised before any rollout — bad configuration should fail loudly and immediately.
  • PolicyError is recorded as a failed trial; whether it aborts the eval is governed by fail_on_error (Inspect semantics).
  • EmbodimentFault and SafetyAbort always halt the eval regardless of fail_on_error — a faulted or unsafe robot must never auto-advance to the next scene unattended.

InspectRobotsError​

class InspectRobotsError(Exception):

Base class for all Inspect Robots errors.

When an error is raised from inside a running trial, the rollout engine attaches the partial TrialRecord — the steps walked and the transcript events up to the failure — as record, so the orchestrator can preserve it in logs. record is None for errors raised outside a rollout (configuration, compatibility, ...).

ConfigError​

class ConfigError(InspectRobotsError):

Invalid task / policy / embodiment configuration. Fail fast.

CompatibilityError​

class CompatibilityError(InspectRobotsError):

A policy and embodiment are not compatible. Fail fast, before any rollout.

PolicyError​

class PolicyError(InspectRobotsError):

The policy raised during inference. Recorded as a failed trial.

Connection-level failures carry a remediation hint in the message.

EmbodimentFault​

class EmbodimentFault(InspectRobotsError):

The embodiment/robot faulted. Always halts the eval and requires a human.

SafetyAbort​

class SafetyAbort(InspectRobotsError):

An approver vetoed an action / e-stop. Always halts the eval.

Adapter conformance​

inspect_robots.conformance​

Adapter conformance: mechanical checks on an embodiment's declared spaces.

Plan 0008 made two consumers of declarations load-bearing: the CLI's default guardrails derive their limits from the action space, and the LLM agent policy builds its whole tool surface from the spaces at bind time. An adapter with missing semantics, missing bounds, unlabeled dims, or a misaligned StateSpec silently degrades both. This module turns those requirements into a checkable report so adapter repos can enforce them in CI (one test: assert_embodiment_conformant(MyEmbodiment().info)) and users can audit an installed adapter via inspect-robots doctor.

check_embodiment is purely declarative and touches no hardware. The separate, opt-in check_guardrail_contribution executes plugin code to validate the optional runtime contribution hook. Conformance proves an adapter is guardrail-ready and agent-ready, not that its declarations are honest (a delta rig declaring absolute-sized per-step bounds type-checks fine). The adapter authoring guide covers the human half.

DEVICE_KINDS​

DEVICE_KINDS = ('v4l2', 'can', 'serial')

DeviceSlot​

class DeviceSlot(arg: str, kind: str, label: str, group: str | None = None):

One device-shaped constructor argument the setup wizard interviews.

arg is the [embodiment.args] key to write; kind selects the probe ("v4l2": /dev/v4l listings with unplug-identify; "can": SocketCAN netdevs from sysfs with unplug-identify; "serial": /dev/serial/by-id listing). label is the human prompt ("left arm CAN channel"). Slots sharing a non-None group are all-or-none: the wizard refuses to write a partial subset of the group.

device_slots​

def device_slots(factory: object) -> tuple[DeviceSlot, ...]:

The declared device slots, defensively read.

Reads DEVICE_SLOTS off factory; anything that is not an iterable of DeviceSlot instances (or contains a slot whose kind is not recognized) has the offending entries ignored, never crashes the wizard. Returns a tuple in declaration order.

OptionSlot​

class OptionSlot(arg: str, label: str, default: bool = False):

One boolean behavior toggle the setup wizard interviews.

arg is the [embodiment.args] key to write (true/false); label is the yes/no question shown to the operator ("Skip the operator start prompts (auto_start)"); default is the suggested answer when the key is absent from an existing config.

option_slots​

def option_slots(factory: object) -> tuple[OptionSlot, ...]:

The declared option slots, defensively read.

Reads OPTION_SLOTS off factory; anything that is not an iterable of OptionSlot instances has the offending entries ignored, never crashes the wizard. Returns a tuple in declaration order.

NumberSlot​

class NumberSlot(arg: str, label: str, default: float | int | None = None, minimum: float | None = None, maximum: float | None = None, allow_none: bool = False):

One numeric constructor argument the setup wizard interviews.

arg is the [embodiment.args] key to write and label is the prompt shown to the operator, including any unit. default is the suggestion when no valid carried value exists. minimum and maximum are inclusive bounds; None leaves that side unbounded. allow_none accepts a disabled value written as none.

number_slots​

def number_slots(factory: object) -> tuple[NumberSlot, ...]:

The declared numeric slots, defensively read.

Reads NUMBER_SLOTS off factory and returns valid NumberSlot instances in declaration order. Non-iterables, malformed entries, non-finite or boolean numeric fields, inverted bounds, and defaults that cannot pass their slot constraints are ignored without crashing setup.

missing_runtime_requirements​

def missing_runtime_requirements(factory: object) -> dict[str, str]:

The declared runtime modules that are not importable here.

Reads RUNTIME_REQUIREMENTS (module name -> remediation command) off factory and probes each with importlib.util.find_spec. Top-level names (the intended use) are probed without executing anything; a dotted name imports its parent package when present, so declare top-level names. ANY probe failure counts as missing (broad except Exception: a present-but-broken parent package propagates arbitrary errors from its __init__, and this checker must never crash setup or doctor). Entries whose key or value is not str, or a RUNTIME_REQUIREMENTS that is not a Mapping, are ignored (a plugin typo must not crash the preflight). Returns the missing subset, insertion-ordered.

ConformanceIssue​

class ConformanceIssue(severity: str, code: str, message: str):

One finding: severity is "error" (fails the check) or "warning".

ConformanceReport​

class ConformanceReport(embodiment: str, issues: tuple[ConformanceIssue, ...] = tuple()):

All findings for one embodiment; ok iff there are no errors.

check_embodiment​

def check_embodiment(info: EmbodimentInfo) -> ConformanceReport:

Check an embodiment's declarations against the plan-0008 requirements.

Errors: missing action semantics; missing/non-finite bounds; missing or duplicate dim_labels; absolute-target modes without exactly one StateSpec field shaped like the action space; a space the default guardrail chain refuses to limit. Warnings: control_hz undeclared (agent motion falls back to 10 Hz step counting) and zero-width bound dims. Purely declarative — safe to run anywhere, no hardware touched.

assert_embodiment_conformant​

def assert_embodiment_conformant(info: EmbodimentInfo) -> None:

Pytest-friendly wrapper: raise AssertionError with the full summary.

check_guardrail_contribution​

def check_guardrail_contribution(embodiment: Embodiment, action_space: Box) -> ConformanceReport:

Validate an optional contribution hook by executing plugin code.

An absent attribute passes. A present attribute must be callable and must return GuardrailContribution. Exceptions raised by the hook propagate as plugin bugs. Unlike check_embodiment, callers must opt in knowing this check can run arbitrary adapter code.

assert_guardrail_contribution_conformant​

def assert_guardrail_contribution_conformant(embodiment: Embodiment, action_space: Box) -> None:

Raise AssertionError with the full contribution report on failure.

Evaluation & logs​

inspect_robots.eval​

The eval() entry point — orchestrates scenes x epochs into an EvalLog.

Mirrors Inspect AI's eval(): it runs a task's scenes (repeated over epochs), scores each recorded trajectory, reduces epochs, aggregates metrics, and returns a list of immutable EvalLog (one per task). The tracer slice accepts already-constructed objects; registry-string resolution (policy="openvla/7b") is layered on with the registry milestone.

eval​

def eval(task: Task | str, policy: Policy | str, embodiment: Embodiment | str, *, log_dir: str = 'logs', sinks: list[LogSink] | None = None, seed: int | None = 0, fail_on_error: bool | float = False, controller: Controller | None = None, approver: Approver | None = None, remap: dict[str, str] | None = None, store_frames: bool = False, store_actions: bool = True, operator_input: OperatorInput | None = None, before_scoring: Callable[[TrialRecord, Scene], None] | None = None, grader: Grader | str | None = None) -> list[EvalLog]:

Run task with policy on embodiment; return [EvalLog].

task/policy/embodiment may be objects or registry names (e.g. policy="scripted"), resolved through the registry — the Inspect-style ergonomic that keeps logs and the CLI reproducible. An embodiment resolved from a registry name is owned by eval() and is closed when the run finishes (even on a halt); a caller-constructed embodiment object stays open — the caller owns its lifecycle.

seed=None draws a fresh seed from the OS and records it in the log, so an "unseeded" run remains reproducible after the fact (and is distinct from seed=0).

fail_on_error follows Inspect semantics for PolicyError (True = fail on first, False = never, 0<x<1 = proportion, x>1 = count), checked after every trial. EmbodimentFault/SafetyAbort always halt regardless. Errored trials are recorded (with their partial trajectory delivered to sinks) but never scored, so a failed trial cannot masquerade as data in the metrics; it stays visible via SceneResult.status and an empty entry in SceneResult.epochs.

A run in which every trial errored (nothing was scored) always ends with status == "error", regardless of fail_on_error.

Ctrl-C during a rollout records the partial trial and writes a log with status == "cancelled", then re-raises the interrupt (as a KeyboardInterrupt subclass chaining the original) after on_eval_end completes. An interrupt outside the rollout call (during scoring, reducers, or log assembly), or a second interrupt during the cancellation handlers, may still prevent the log from being written.

When store_frames is set, camera frames are streamed to <log_dir>/frames as binary side-cars (R5) rather than kept in memory.

store_actions defaults to True and writes each trial's complete executed action sequence to <log_dir>/actions as an atomic JSONL side-car. The actions trial-metadata key is framework-reserved. Set store_actions to False to disable these files.

operator_input supplies attended-console input; None disables the channel.

Before a grader runs, an embodiment's optional duck-typed observe_parked() hook may move the robot to its parked/rest pose so the cameras see the scene unobstructed and return one fresh Observation. Returning None declines. Other failures degrade with a RuntimeWarning and grading uses the last-step frames, except SafetyAbort and EmbodimentFault, which halt the eval.

before_scoring is called exactly once per trial that will be scored (never for errored or cancelled trials, which are recorded but not scored), after the rollout returns and before the scorers run. It may mutate the record — e.g. capture TrialRecord.operator_judgement (R6) so the operator scorer can read it, and TrialRecord.operator_note alongside it, which is recorded but never scored. Exceptions it raises propagate to the caller. Note this fires on the other side of scoring from LogSink.on_trial_end.

grader is the component form of the same seam: a Grader object or registry name whose grade method becomes the pre-scoring hook. Pass either grader or before_scoring, not both (ConfigError); disable grading with grader=None (the registry name "none" is CLI vocabulary, not an API value).

Raises CompatibilityError (fail fast, before any rollout) if the policy and embodiment are incompatible, and ConfigError for an invalid epoch reducer.

eval_set​

def eval_set(tasks: Task | str | Sequence[Task | str], policy: Policy | str, embodiment: Embodiment | str, *, log_dir: str = 'logs', sinks: list[LogSink] | None = None, seed: int | None = 0, fail_on_error: bool | float = False, controller: Controller | None = None, approver: Approver | None = None, remap: dict[str, str] | None = None, store_frames: bool = False, store_actions: bool = True, operator_input: OperatorInput | None = None, before_scoring: Callable[[TrialRecord, Scene], None] | None = None, grader: Grader | str | None = None, retry_attempts: int = 0) -> tuple[bool, list[EvalLog]]:

Run a set of tasks and return (success, logs) (mirrors Inspect AI).

success is True iff every returned log has status == "success". A task that raises before or without producing a log contributes one status="error" log carrying the exception text, and the remaining tasks still run. A SafetyAbort or EmbodimentFault that escapes eval() (raised outside a trial) and KeyboardInterrupt still propagate. A halt inside a trial ends that task with an error log and the set continues to the next task. CompatibilityError, unknown policy or embodiment registry names, and task-factory ConfigError are therefore reported once per affected task. Only grading configuration errors raised before the task loop propagate.

With a string embodiment, a non-safety exception from close() can produce an error row even when that task's completed JSON log is already on disk.

store_actions follows eval()'s default-on action side-car contract.

grader/before_scoring follow eval()'s contract (one pre-scoring hook, not both) and are resolved once here, so every task shares the same grader instance.

Caller-supplied sinks are reused across the set's sequential runs. Each sink must reset its per-run state in on_eval_start and tolerate one complete lifecycle per task.

Resumption of a partially-completed run (skipping already-finished scenes via a stable run id) is reserved for a follow-up: retry_attempts is accepted now so callers don't get retrofitted, but is not yet honored.

inspect_robots.log​

The immutable evaluation log — Inspect Robots's reproducible record of a run.

Mirrors Inspect AI's EvalLog: version + status + eval spec + results + stats + per-scene samples + error. Serialized to JSON with a schema version so newer Inspect Robots always reads older logs (a read-back guarantee enforced by golden tests in a later step).

Immutability is shallow: the dataclasses are frozen and sequence fields are tuples, so reassigning a field or mutating the sample list is impossible — but dict-valued fields (SceneResult.reduced, the per-epoch score dicts, EvalResults.metrics, EvalSpec.policy_config / embodiment_info, and SceneResult.scene_metadata) remain plain mutable dicts, as do the dictionaries inside SceneResult.operator_messages. SceneResult.policy_transcripts entries are arbitrary mutable JSON values. Treat a log as read-only; nothing deep-freezes it.

SCHEMA_VERSION​

SCHEMA_VERSION = 1

EvalSpec​

class EvalSpec(task: str, policy: str, embodiment: str, created: str, inspect_robots_version: str, git_commit: str | None = None, policy_config: dict[str, Any] = dict(), embodiment_info: dict[str, Any] = dict(), seed: int | None = None, max_steps: int | None = None, max_seconds: float | None = None):

Top-level identity and configured horizon of a reproducible eval.

max_steps is always the resolved integer budget used by the rollout. max_seconds preserves a benchmark's declared physical-time budget when that integer was derived from the embodiment's control rate.

EvalStats​

class EvalStats(started_at: str, completed_at: str, duration_s: float, total_steps: int, mean_inference_latency_s: float | None = None, frames_dir: str | None = None):

Timing and execution statistics for a run.

SceneResult​

class SceneResult(scene_id: str, status: str, reduced: dict[str, float] = dict(), epochs: tuple[dict[str, float], ...] = (), error: str | None = None, instruction: str | None = None, scene_metadata: dict[str, Any] = dict(), operator_judgements: tuple[str | None, ...] = (), judgement_sources: tuple[str | None, ...] = (), operator_notes: tuple[str | None, ...] = (), operator_messages: tuple[tuple[dict[str, Any], ...], ...] = (), trial_metadata: tuple[dict[str, Any], ...] = (), termination_reasons: tuple[str | None, ...] = (), policy_transcripts: tuple[Any, ...] = ()):

Persist one scene's metadata, reduced scores, and raw per-epoch records.

EvalResults​

class EvalResults(total_scenes: int, total_trials: int, metrics: dict[str, float] = dict(), errored_trials: int = 0):

Aggregate results across all scenes.

EvalLog​

class EvalLog(version: int, status: str, eval: EvalSpec, results: EvalResults, stats: EvalStats, samples: tuple[SceneResult, ...] = (), error: str | None = None):

The full record returned by eval and persisted to disk.

read_eval_log​

def read_eval_log(path: str) -> EvalLog:

Read an EvalLog back from a JSON file on disk.

Logging sinks​

inspect_robots.logging.sink​

The LogSink protocol, its optional extension, and a no-op base implementation.

A sink observes a run's lifecycle. The rollout engine and eval() call these hooks in a fixed order: on_eval_start → (per trial: on_trial_start → log_step* → on_trial_end) → on_eval_end.

Sinks may additionally define the duck-typed log_policy_messages(t, messages) extension. The rollout calls it at most once per control step and only when the policy performed an inference. Policy implementations are expected to supply plain-JSON-type messages shaped like TrialRecord.policy_transcript entries, but core does not enforce or normalize that shape on this live path, so sinks must render defensively. Sinks must not mutate the supplied messages. The extension deliberately stays off both LogSink and NullSink: it must not change structural protocol conformance or advertise a no-op that makes policies build transcript deltas.

A second duck-typed extension, bind_spaces(action_space, observation_space), is called by eval() exactly once per run, before on_eval_start, with the resolved embodiment Box and ObservationSpace. A sink that defines a callable bind_spaces MUST accept this two-argument signature; like log_policy_messages, the name is claimed by the eval loop.

A third duck-typed extension, bind_frames_dir(frames_dir), is called by eval() exactly once per run, after bind_spaces and before on_eval_start. The argument is the run's stored-frame directory as a string, or None when frame storage is off. A callable with this claimed name MUST accept that one-argument signature. It deliberately remains off LogSink and NullSink like the other optional extensions.

LogSink​

class LogSink(Protocol):

Observes the lifecycle of an evaluation run.

NullSink​

class NullSink:

A sink that does nothing — a convenient base for partial implementations.

inspect_robots.logging.json_log​

The canonical JSON eval-log sink.

Writes the immutable EvalLog to log_dir once the run finishes. The write is atomic (temp file + os.replace) so an interrupted overnight run never leaves a half-written log.

The file is strict RFC 8259 JSON: non-finite floats (nan, ±inf, e.g. a min_distance_to_goal score when no distance was ever recorded) are mapped to null before serialization, so any conforming parser (jq, browsers, non-Python tooling) can read the log. allow_nan=False is kept on the json.dump call as a regression backstop: if a non-finite value ever slips past the sanitizer, writing fails loudly instead of emitting Infinity/ NaN literals.

JsonLogSink​

class JsonLogSink(log_dir: str):

Persist the final EvalLog as JSON.

Per-step data lives in the TrialRecord/FrameStore, not here; this sink only writes the final log (path holds where it landed).

inspect_robots.logging.live_log​

Maintain an atomic, schema-valid snapshot while an evaluation is running.

LiveLogSink​

class LiveLogSink(log_dir: str, *, min_write_interval_s: float = 1.0, clock: Callable[[], float] = time.monotonic):

Persist reusable in-progress snapshots without becoming the final-log sink.

One instance may be reused across sequential runs: on_eval_start clears all prior state and chooses a new path. Pair this sink with JsonLogSink when a final log is required. A sinks list containing only LiveLogSink leaves no log after a successful run because on_eval_end removes the live snapshot.

inspect_robots.logging.rerun_sink​

Optional Rerun visualization sink.

Logs camera images, proprioception, action vectors, and success markers to Rerun <https://github.com/rerun-io/rerun>_. The sink can write a .rrd recording, spawn a local viewer, connect over gRPC to a remote viewer, or tee the same stream to a file and live viewer with rerun-sdk 0.24 or newer. A viewer spawned by the sink has a 2 GiB memory limit by default, which makes the viewer purge its oldest events instead of accumulating an unbounded session history. rerun-sdk is imported lazily inside methods so the core package never depends on it; if it is not installed, the sink warns once and becomes a no-op (so unattended runs and the core-only import gate are unaffected).

Emission happens on a daemon worker thread: log_step snapshots the transition and enqueues it, so a slow or stalled viewer connection can never block the control-rate rollout loop. Under backpressure the sink degrades visualization instead of delaying control: camera frames are dropped first (scalar plots stay complete), then whole steps or transcript rows, and the drop counts are reported as a RuntimeWarning when the eval ends. The queue is drained at every trial boundary (bounded by flush_timeout), so an eval that aborts mid-run loses at most the current trial's queued tail. Camera frames are JPEG-compressed by default (jpeg_quality=75); pass jpeg_quality=None for lossless raw frames. If compression is unavailable (an SDK without Image.compress, or pillow missing), the sink warns once and logs raw frames. Caller-path startup performs init, spawn, connect_grpc, save, set_sinks, GrpcSink/FileSink construction, and the recording_dir mkdir. All later Rerun SDK calls happen on the worker because the SDK's timeline state is thread-local, except for the shutdown-path flush probe and unregister_shutdown on the caller path. The probe invokes RecordingStream.flush on a bounded daemon thread; flush is internally synchronized, and unregister_shutdown only manipulates an atexit hook, so neither depends on timeline state. Worker state is generation-scoped so a worker wedged in a blocked SDK call is disowned at shutdown and can never double-consume after a restart. If the SDK flush probe also wedges, the sink unregisters the SDK's unbounded atexit flush, disables itself, and abandons queued SDK-side data.

The viewer limit applies only to viewers this package spawns; a viewer already running on the same port keeps the limit it started with. The bounded exit probe requires rerun-sdk 0.22 or newer because older recording streams expose no flush method. A new sink in the same process can still hang in rr.init after a connection wedges, and paths such as Ctrl-C that skip on_eval_end retain the SDK's unbounded atexit hook.

Each trial's entities are namespaced under trial/<scene_id>/e<epoch> so successive trials never overwrite one another on the shared step timeline. The worker sends an explicit per-trial blueprint from _emit for the live namespace. Transcripts are emitted as TextLog rows at {prefix}/llm paired with a markdown TextDocument at {prefix}/llm/latest holding the step's assistant message(s) for a wrapped, timeline-synced reading pane.

Install with pip install "inspect-robots[rerun]".

RerunSink​

class RerunSink(recording_path: str | None = None, *, recording_dir: str | None = None, application_id: str = 'inspect_robots', spawn: bool = False, spawn_memory_limit: str = '2GiB', spawn_port: int = 9876, connect_url: str | None = None, jpeg_quality: int | None = 75, queue_size: int = 64, flush_timeout: float = 10.0):

Record and visualize one Rerun stream without delaying the control loop.

A recording target can be combined with either live mode when rerun-sdk 0.24 or newer provides set_sinks. recording_dir creates one file per eval named {task_slug}_{eight_hex_digits}.rrd; recording_path reuses one fixed file. resolved_recording_path exposes the file attached for the current eval, or None when no file was attached. For a reused sink, the next eval's rr.init releases the previous FileSink after on_eval_end has probed its flush. A fixed path is truncated and rewritten on each eval, matching rr.save semantics.

spawn_memory_limit is passed verbatim to Rerun only when spawn=True.

Registry & CLI​

inspect_robots.registry​

Registry and decorators for framework extension components.

Mirrors Inspect AI's extension model: components register by name via decorators and are resolved from strings (so eval(policy="scripted") and the CLI work). Out-of-tree packages publish components through importlib.metadata entry-point groups, so an installed inspect-robots-openvla appears in inspect-robots list without being imported first.

Entry-point groups: inspect_robots.tasks, inspect_robots.policies, inspect_robots.embodiments, inspect_robots.scorers, inspect_robots.graders, inspect_robots.sinks, inspect_robots.operator_inputs.

Kind​

Kind = str

KINDS​

KINDS: tuple[Kind, ...] = ('task', 'policy', 'embodiment', 'scorer', 'grader', 'sink', 'operator_input')

F​

F = TypeVar('F', bound=(Callable[..., Any]))

register​

def register(kind: Kind, name: str | None = None) -> Callable[[F], F]:

Register a factory under kind/name (defaults to its __name__).

task​

def task(name: str | None = None) -> Callable[[F], F]:

Decorator: register a task factory under name.

policy​

def policy(name: str | None = None) -> Callable[[F], F]:

Decorator: register a policy factory under name.

embodiment​

def embodiment(name: str | None = None) -> Callable[[F], F]:

Decorator: register an embodiment factory under name.

scorer​

def scorer(name: str | None = None) -> Callable[[F], F]:

Decorator: register a scorer factory under name.

grader​

def grader(name: str | None = None) -> Callable[[F], F]:

Decorator: register a grader factory under name.

sink​

def sink(name: str | None = None) -> Callable[[F], F]:

Decorator: register a log-sink factory under name.

operator_input​

def operator_input(name: str | None = None) -> Callable[[F], F]:

Decorator: register an operator-input factory under name.

registered​

def registered(kind: Kind) -> dict[str, Callable[..., Any]]:

Return all registered factories for kind (builtins + plugins).

resolve​

def resolve(kind: Kind, name: str, /, **kwargs: Any) -> Any:

Construct a registered component by name with the given keyword args.

inspect_robots.cli​

The inspect_robots command-line interface.

Subcommands:

  • inspect-robots list [tasks|policies|embodiments|scorers|sinks|operator_inputs] — show registered components (builtins + installed plugins).
  • inspect-robots run --task T --policy P --embodiment E — run an eval, resolving components from the registry. --instruction runs an operator-written ad-hoc task, while --auto-task and repeatable -A k=v generate one from the initial camera frames. Pass constructor args with -T/-P/-E k=v; --epochs, --fail-on-error, and --store-frames tune the run. The written log's path is printed at the end.
  • inspect-robots eval-set TASK [TASK ...] --policy P --embodiment E — run several registered tasks (exact names or fnmatch globs, e.g. 'kitchenbench/*') against one resolved policy/embodiment pair via eval_set. Prints one status line and a compact per-task row instead of a full summary per task.
  • inspect-robots inspect LOG.json [--transcript] [--wire [CALL]] — print a saved eval log and optionally append policy conversations or captured wire calls.
  • inspect-robots summarize LOG.json [--model M] — distill a saved eval log into a deterministic digest or model-written learnings file.
  • inspect-robots view LOG.json|LOG_DIR [-o PATH] [--open] [--serve] — render one saved eval log or a browsable directory index as self-contained HTML, optionally serving a directory until stopped.
  • inspect-robots video LOG.json — render a --store-frames run's stored camera frames to one MP4 per (trial, camera) stream via the ffmpeg binary.
  • inspect-robots setup — interactively configure defaults and camera devices.

Zero-config form (plan 0005): inspect-robots "place the spoon on the plate" is sugar for run --instruction "..." — a single ad-hoc scene on the user's default policy/embodiment (flags > INSPECT_ROBOTS_POLICY/_EMBODIMENT env vars > ~/.config/inspect-robots/config.ini). The sugar only fires for a first argument with interior whitespace, so a mistyped subcommand (inspect-robots isnpect) errors instead of starting a robot rollout; single-word instructions use the explicit run --instruction form.

DEFAULT_RERUN_CONNECT_URL​

DEFAULT_RERUN_CONNECT_URL = 'rerun+http://127.0.0.1:9876/proxy'

build_parser​

def build_parser() -> argparse.ArgumentParser:

Build the command-line parser and its subcommands.

main​

def main(argv: Sequence[str] | None = None) -> int:

Parse arguments, dispatch one subcommand, and return its process exit code.

Mock world​

inspect_robots.mock.cubepick​

CubePick — a deterministic 2D toy world for exercising the full stack.

A point end-effector in the unit square must reach a cube. The action is a 2D end-effector position delta. Success is declared (and exposed as privileged info["success"]) when the effector is within goal_radius of the cube. Fully deterministic given a seed; no third-party dependencies.

CubePickEmbodiment​

class CubePickEmbodiment(*, max_step: float = 0.1, goal_radius: float = 0.05, start: tuple[float, float] = (0.1, 0.1)):

A 2D reach-the-cube simulator.

inspect_robots.mock.policies​

Mock policies for the CubePick world.

  • ScriptedPolicy — a deterministic oracle that walks the effector to the cube. It predicts a full action chunk by simulating its own future motion, so the chunk is a genuine open-loop trajectory (H > 1).
  • RandomPolicy — emits random deltas; mostly fails.
  • NoopPolicy — emits zero actions; never succeeds.

ScriptedPolicy​

class ScriptedPolicy(*, chunk_size: int = 4, max_step: float = 0.1):

Deterministic oracle: walk straight to the cube, in chunks.

RandomPolicy​

class RandomPolicy(*, chunk_size: int = 4, max_step: float = 0.1, seed: int = 0):

Emit random small deltas. Deterministic given the construction seed.

NoopPolicy​

class NoopPolicy(*, chunk_size: int = 1):

Emit zero actions; never moves.