rewardkit

The full skill.

Original instructions from the publisher’s SKILL.md

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python 
package that turns a directory of criteria files into a reward score. Each criterion is a 
Python function call or a TOML judge file; folders become separate rewards.

## Setup in a Harbor task

Put criteria alongside `test.sh` in the task's `tests/` directory:

```
tests/
├── test.sh
├── checks.py         # programmatic criteria
└── judge.toml        # optional LLM/agent judge
```

`tests/test.sh`:
```bash
#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests
```

This runs all criteria in `/tests/` against the workspace at `/app` and writes 
`/logs/verifier/reward.json`. Defaults match Harbor's conventions — no extra config needed.

If judge criteria need API keys, pass them through `task.toml`:
```toml
[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
```

Ask whether Reward Kit should run in the agent's shared environment or in a
separate verifier environment. Prefer a separate verifier environment when judge
prompts, grading dependencies, API keys, or clean-room checks should not be
available to the agent:

```toml
[environment]
network_mode = "no-network"   # Agent env baseline — offline during agent.run()

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"     # Verifier env baseline — LLM judge API calls
docker_image = "python:3.12-slim"
```

In shared mode, the verifier runs in the agent container and inherits
`[environment].network_mode`. Put `[verifier].network_mode` only when verify()
needs different network access than the agent phase (a phase override, not a
baseline). If agent and verifier need different baselines without runtime
switching, use `environment_mode = "separate"` and set
`[verifier.environment].network_mode`.

Judge criteria that call external APIs need a `public` baseline or allowlist on
the verifier environment. Programmatic checks that only read local files can use
`no-network`.

In separate mode, `tests/` is the verifier image build context and must provide
`/tests/test.sh` at runtime; Harbor does not upload `tests/` into the running
verifier container.

## Programmatic criteria

Call built-ins from any `.py` file in `tests/`:

```python
import rewardkit as rk

rk.file_exists("output.txt")
rk.file_contains("output.txt", "hello")
rk.command_succeeds("python main.py", weight=2.0)
rk.json_key_equals("result.json", "status", "ok")
```

All criteria accept `weight` (default `1.0`) and `isolated` (default `False`, runs in 
overlayfs so side effects don't leak).

### Available built-ins

- **Files**: `file_exists`, `file_not_exists`, `file_contains`, `file_contains_regex`, 
  `file_matches`, `files_equal`, `diff_ratio`
- **Commands**: `command_succeeds`, `command_output_contains`, `command_output_matches`, 
  `command_output_matches_regex` (30s default timeout, optional `cwd`)
- **Data**: `json_key_equals`, `json_path_equals`, `csv_cell_equals`, `xlsx_cell_equals` 
  (needs `[office]` extra), `sqlite_query_equals`
- **HTTP**: `http_status_equals`, `http_response_contains`
- **Images**: `image_similarity`, `image_size_equals` (needs `[image]` extra)
- **Trajectory**: `trajectory_tool_used`, `trajectory_tool_not_used`, `trajectory_turn_count`

For extras, install with `uv tool install harbor-rewardkit[all]`.

## Custom criteria

Use the `@criterion` decorator. First parameter is always `workspace: Path`. Returns 
`bool` or `float`:

```python
from pathlib import Path
from rewardkit import criterion

@criterion
def has_valid_output(workspace: Path) -> bool:
    return (workspace / "output.txt").read_text().strip() != ""
```

Zero-parameter criteria auto-register. Criteria with extra args must be called via `rk`:

```python
@criterion(description="output has at least {n} lines")
def has_n_lines(workspace: Path, n: int) -> bool:
    return len((workspace / "output.txt").read_text().splitlines()) >= n

rk.has_n_lines(10, weight=2.0)
rk.has_n_lines(50, weight=1.0)
```

For criteria shared across reward subdirs, define with `shared=True` in a root-level file 
and call from subdirs.

## Judge criteria (LLM or agent-as-a-judge)

For subjective checks (quality, readability, edge cases), create a TOML file:

```toml
[judge]
judge = "anthropic/claude-sonnet-5"   # LiteLLM model string
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0
```

Criterion types:
- `binary` — yes/no → 1.0 or 0.0
- `likert` — 1..points, normalized to [0, 1]
- `numeric` — min..max, normalized to [0, 1]
- `rubric` — 2 to 10 described `levels` forming a scale from worst to best; position sets the score

### Agent judges

Agent judges shell out to a CLI and can explore the filesystem:

```toml
[judge]
judge = "claude-code"
model = "anthropic/claude-sonnet-5"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"
```

Slower and more expensive than LLM judges, but they can run commands and inspect files.

### JEV judge

JEV is a new type of language model from TypeSafe. It answers each criterion with a probability or a rubric
score and returns no reasoning, so it is fast and cheap. It needs the `jev` extra
(`harbor-rewardkit[jev]`) and `TYPESAFE_API_KEY`.

```toml
[judge]
judge = "jev"
files = ["/app/answer.md"]

[[criterion]]
description = "Does the answer address the requested task?"

[[criterion]]
description = "How complete is the answer?"
type = "rubric"
levels = ["Omits the information", "Covers part of it", "Covers all of it"]
```

Binary criteria pass at a probability of 0.5 or higher. Only `binary` and `rubric` criteria and
text files are supported; `atif-trajectory` and `prompt_template` are not.
The task image needs CA certificates (`ca-certificates` on Debian and Ubuntu), because the
TypeSafe SDK verifies TLS against the system trust store.

### Useful `[judge]` options

`timeout` (default 300), `reasoning_effort` (`low`|`medium`|`high`), `reference` (path to 
reference solution), `atif-trajectory` (evaluate the agent's trajectory), `weight`, 
`prompt_template` (custom prompt with `{criteria}` placeholder).

### Scoring aggregation (within one judge TOML)

```toml
[scoring]
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7             # only for threshold
```

Only affects how this file's own criteria combine. To aggregate *across*
dimensions, see [Aggregating dimensions](#aggregating-dimensions).

### Scoring config for programmatic files

Each `.py` file that registers criteria is an equal-weighted scoring component
named after its filename stem. Files that only provide imports or shared
criterion factories and register no checks are ignored. To change how criteria
within a file combine, use `[scoring.<stem>]` in the same directory's
`reward.toml`:

```toml
# tests/structure/reward.toml
[scoring.files_exist]       # configures files_exist.py
aggregation = "all-pass"

[scoring.behavior]          # configures behavior.py
aggregation = "threshold"
threshold = 0.75
```

Each entry takes the same aggregation values as a judge TOML. Unknown keys and
stems that do not resolve to a criterion-bearing Python file raise.

Directories may be nested recursively. A non-root directory can aggregate its
local Python files, local judges, and immediate child directories with one
unnamed `[[reward]]` table:

```toml
# tests/correctness/reward.toml
[[reward]]
aggregation = "weighted-mean"
weights = { files = 2.0, behavior = 1.0 }
```

Membership is implicit. Child directories have weight 1.0 unless overridden;
use filename stems for local Python files and judge TOMLs, and directory names
for child groups. Without `[[reward]]`, the directory defaults to weighted mean.

## Multi-reward tasks

Put criteria in subdirectories — each becomes a separate reward:

```
tests/
├── test.sh
├── correctness/
│   └── check.py
├── structure/
│   └── files_exist.py
└── quality/
    └── quality.toml
```

Judge TOMLs may also sit directly at the tests root alongside reward
subdirectories. Each is exposed as a top-level reward named after its filename
stem and can be referenced by a root aggregation.

Criterion-bearing Python files at the tests root are also top-level dimensions
named after their stems. Root support files that register no criteria are
ignored.

Produces:
```json
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6 }
```

### Aggregating dimensions

To add aggregated scores on top of the per-dimension keys, add a root-level
`tests/reward.toml` with one or more `[[reward]]` tables. Each adds one key to
`reward.json`, aggregating the dimensions with the same modes as `[scoring]`:

```toml
# tests/reward.toml
[[reward]]
name = "reward"
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
# threshold = 0.7          # only for threshold
weights = { correctness = 2.0, quality = 1.0 }
```

```json
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6, "reward": 0.0 }
```

The per-dimension scores stay; aggregated keys are added alongside them (a
`name` may not collide with a dimension). Top-level dimensions have equal
weight unless that aggregation's inline map overrides them;
`reward-details.json` keeps the full recursive breakdown.

## Output files

- `/logs/verifier/reward.json` — per-reward scores
- `/logs/verifier/reward-details.json` — per-criterion results, judge reasoning, errors

## Multi-step tasks

In a multi-step task, each step has its own `tests/` under
`steps/{name}/tests/`, and the verifier runs once per step. Reward Kit behaves
the same as in a single-step task: for each step it reads `/tests`, runs the
criteria against `/app`, and writes `/logs/verifier/reward.json` for that step.
Harbor then aggregates per-step results into a trial-level reward via
`multi_step_reward_strategy` in `task.toml` — aggregation happens *outside*
Reward Kit, so don't try to encode cross-step logic in your criteria.

A task-level `tests/` directory (at the task root) is uploaded to `/tests`
first, then the step's own `tests/` is layered on top (same-name files win).
Put shared helpers (common `checks.py` functions with `shared=True`, fixture
files, a fallback `test.sh`) at the task level, and step-specific criteria
under each step.

Multi-reward subdirectories still work *within* a step: `steps/foo/tests/`
can contain `correctness/`, `structure/`, `quality/` — each produces a
separate reward key for that step, and `multi_step_reward_strategy = "mean"`
averages each key across steps. Use `"final"` when the last step is an
end-to-end check whose rewards already represent the full task.

## When to reach for what

- **Use built-ins** for file existence, string matches, command output, JSON/CSV checks, 
  HTTP probes.
- **Use `@criterion`** when logic is task-specific but still programmatic.
- **Use LLM judges** for subjective quality dimensions (readability, correctness of prose).
- **Use agent judges** when the rubric requires exploring the filesystem or running code 
  (e.g. "does the test suite actually pass?").
- **Use subdirectories** when you want separate scores (correctness vs structure vs 
  quality) rather than one blended number.
- **Use `isolated=True`** for any criterion that runs mutating commands, so it doesn't 
  corrupt the workspace for other criteria.

## Working example

See `examples/tasks/reward-kit-example/` in the Harbor repo.