eval

The full skill.

Original instructions from the publisher’s SKILL.md

Run one evaluation of a Library skill and show its board. Eval never mints a version and never moves skill bytes. When the evaluated bytes are already a published version it publishes that run's receipt to the team (`--no-commit` keeps it on this machine); otherwise it prints a share hint.

Arguments: everything after the command (in Claude Code this arrives as "$ARGUMENTS"): one skill name (or nothing when the working directory is inside a Library skill folder) or several, then any of `--k <n>`, `--triggers-only`, `--execution-only`, `--case <stem>`, `--model <m>`, `--judge-model <m>`, `--no-gen`, `--no-commit` (keep the receipt on this machine instead of publishing it to the team), `--parallel <n>`, `--pending` (every shared skill with no receipt for its current version), `--window overnight|later` (queue instead of running; nothing is paid), passed through unchanged. Leave `--batch <n>` out: it asks "Continue?" before each further batch, which needs a terminal. The queue flags (`--queue-list`, `--dequeue <skill>`, `--drain …`) belong to the terum-skills skill; `--dequeue` and `--drain` need a confirmation first.

## Command

    npx -y terum-skills@latest eval <skill> [flags] --format md

Run it in the background when your shell tool can, or with a long timeout: a full matrix takes minutes. Show the board when it finishes.

## Before

Confirm with the user first: trials and generation run their logged-in Claude Code and bill their account. State what the run buys — cases × k × arms (k defaults to 1; two arms, three when a published incumbent exists) plus generation calls for any missing cases or triggers, times the number of skills for a batch (several skills run as one batch after one preflight, `--parallel <n>` at a time, default four; confirm once for the whole batch) — and, when the skill-info board shows a previous run, quote its cost and duration as the estimate. `--window overnight|later` queues instead of running and costs nothing now. Suggest `--triggers-only` or `--case <stem>` for a first look. What eval does, so the user knows:

1. Finds the local folder by name; it need not belong to a team. When a team is configured, the clone is fetched for policy and incumbent history, never for the candidate bytes.
2. Runs hygiene on the local bytes, allowing absent managed fields; errors stop before any agent call.
3. Preflight checks `claude --version` and a one-turn smoke task before paid trials.
4. Uses existing cases and triggers, generating only the missing assets for the requested tiers; it prints the target path and the content-change consequence before writing them into the folder. Generating assets changes content identity and can mint a version on the next publish; `--no-gen` uses only existing assets, and regenerating cases means deleting `evals/cases/` and re-running. The generated bytes are re-checked against HYG2/HYG3 (the same predicates and author exemption the next command uses) before the write-back; a failure there refuses the run and writes nothing.
5. Runs trigger evals and execution cases, each k times, comparing the candidate with the baseline and, when available, a digest-different published incumbent chosen by receipt recency.
6. Stores receipts and run artifacts under `~/.terum/skills/evals/local/<digest>/<run-id>/` and prints the report; a later publish attaches receipts whose digest matches the published bytes; when these exact bytes are already a published version the receipt goes to the team now (`--no-commit` keeps it local).

## After

Show the board verbatim: the verdict band, the comparison table and any `⚠` marks stay as printed; never average holes into a clean number. `Already evaluated these exact bytes` is a result — the earlier receipt stands. A batch shows one board with `Evaluated X of N` and the queued remainder; offer the eval-report skill for each skill afterwards. Model flags pass through unchanged (`--model`, `--judge-model`; default `sonnet`). Editing the skill changes its digest, so the Library's score belongs to the bytes that were evaluated; installed receipts keep their original runner and are not proof this user ran anything. The one exit-1 that does ask for a re-run is the generated-assets hygiene refusal, which says so in its own text.

## Rules

- No TTY: never pipe `y`, never drive the CLI with `expect`, never add `--frames` to dodge a question. A question the CLI asks means the verb belongs in a terminal: say so and hand over the command.
- Never use the skill-file `` !`command` `` injection; run every command with your shell tool and read its output.
- Do not `cd`; run from the current working directory and pass absolute paths.
- Exit 1 is a result, not a retry: show the failure block, do not re-run, and do not claim earlier steps were rolled back.

## Sandbox

When `CODEX_SANDBOX_NETWORK_DISABLED=1` is set and the command starts with `npx`, add `--prefer-offline` after `npx` so a cached package resolves without the registry; if npx still reports a network error, ask the user to run the command in a terminal. In that sandbox the verbs that need the network — `sync`, `install`, `publish`, `invite`, `eval`, and `update`'s release probe — are handed to a terminal with the reason.