run-full-suite

The full skill.

Original instructions from the publisher’s SKILL.md

## Purpose

`run-full-suite` scans every `specs/*/spec-integration-tests/` directory for a `run-all.sh` and/or `tc-*.sh` files, runs each suite in sequence, and produces a single consolidated report across all specs. This is the project-wide integration test runner equivalent of `dotnet test` for the speckit shell test layer.

---

## User Input

```text
$ARGUMENTS
```

- Empty: run every discovered spec suite
- Spec filter (e.g. `004`, `token`, `catalog`): run only suites whose parent folder name contains the filter (case-insensitive)
- `--list`: print all discovered suites with TC counts, do not run
- `--fail-fast`: stop after the first spec suite that has any FAIL result

---

## Execution Steps

### Step 1 — Discover all spec suites

From the repo root, find all feature scripts directories:

```bash
find specs -maxdepth 3 -name "run-all.sh" | sort
```

For each found `run-all.sh`, derive:
- `SPEC_DIR` — the parent `specs/{NNN}-{name}/` folder
- `SPEC_ID` — the `{NNN}-{name}` folder name
- `TC_COUNT` — number of `tc-*.sh` files in the same `spec-integration-tests/` directory
- `HAS_CONFIG` — whether `config.env` exists in the scripts directory

If no `run-all.sh` files are found anywhere under `specs/`, stop and tell the user:
"No runnable spec suites found. Run `/speckit-scripttest` on one or more features first."

Apply the `$ARGUMENTS` filter: keep only suites whose `SPEC_ID` contains the filter string (case-insensitive). If the filter matches nothing, tell the user which suites exist and stop.

### Step 2 — Handle `--list`

If `$ARGUMENTS` contains `--list`:

Print a discovery table and stop without running:

```
Discovered speckit test suites:

  #   Spec                           TC Scripts   config.env   Runnable
  ─── ────────────────────────────── ──────────── ──────────── ────────
  1   004-token-wallet-service        27           ✓            ✓
  2   005-token-adjusted-checkout      0           ✗            ✗ (no tc-*.sh yet)
  …

Run '/run-full-suite' to execute all runnable suites.
```

### Step 3 — Check shared prerequisites

Before running any suite, verify tools are available:

```bash
command -v curl  || echo "WARN: curl not found — HTTP tests will fail across all suites"
command -v jq    || echo "WARN: jq not found — JSON assertions will fail across all suites"
command -v psql  || echo "WARN: psql not found — DB tests will fail across all suites"
command -v docker || echo "WARN: docker not found — detect-ports.sh (env var refresh) will not be able to derive PG_CONN"
```

Print a one-line warning for any missing tool. Do not block execution — scripts will report their own FAILs.

### Step 4 — Run each suite

For each suite in discovery order (filtered if `$ARGUMENTS` specified):

1. Print a suite header:
   ```
   ════════════════════════════════════════════════════════
   Suite: 004-token-wallet-service  (27 scripts)
   ════════════════════════════════════════════════════════
   ```

2. **Refresh env vars before running.** If `specs/{SPEC_ID}/spec-integration-tests/detect-ports.sh` exists, run it with `--update` first so `config.env` reflects the current AppHost run's ports and connection strings (Aspire reassigns ports/passwords on every restart, so a stale `config.env` will cause spurious failures):
   ```bash
  bash "specs/{SPEC_ID}/spec-integration-tests/detect-ports.sh" --update
   ```
   If the script reports it cannot detect a running project (e.g. the AppHost isn't running), surface that warning to the user before continuing — the suite will likely fail without it.

3. Run the suite using `run_in_terminal`:
   ```bash
  bash "specs/{SPEC_ID}/spec-integration-tests/run-all.sh"
   ```

4. Parse stdout for the `TEST SUMMARY` block — extract `Passed`, `Failed`, `Skipped`, `Total`.

5. Store results as `{SPEC_ID, passed, failed, skipped, total, duration_seconds}`.

6. If `--fail-fast` is set and `failed > 0`: print the failure details for this suite, then stop and print the partial cross-spec summary (Step 5) with a note that execution was halted early.

7. Continue to the next suite regardless of failures (unless `--fail-fast`).

### Step 5 — Cross-spec summary report

After all suites have run, print the consolidated report:

```
════════════════════════════════════════════════════════════════════
  FULL SUITE REPORT — {ISO datetime}
════════════════════════════════════════════════════════════════════

  Spec                           PASS   FAIL   SKIP   TOTAL   Status
  ─────────────────────────────  ─────  ─────  ─────  ─────   ──────
  004-token-wallet-service        25     2      0      27      ✗ FAIL
  005-token-adjusted-checkout      —     —      —       —      ○ NO SCRIPTS
  …

  ─────────────────────────────────────────────────────────────────
  TOTALS                          {N}   {N}    {N}    {N}
  Overall status: {PASS / FAIL}
════════════════════════════════════════════════════════════════════
```

**Status key**:
- `✓ PASS` — all TCs in this suite passed
- `✗ FAIL` — one or more TCs failed (list them below the table)
- `○ NO SCRIPTS` — directory exists but has no `tc-*.sh` yet
- `⚠ SKIPPED` — suite was excluded by the filter argument

**Failure detail block** (printed below table if any failures):

```
Failures:
  004-token-wallet-service
    ✗ tc-022.sh — FAIL: net_balance expected=0 actual=80
    ✗ tc-043.sh — FAIL: HTTP 409 expected, got 200
```

### Step 6 — Exit guidance

- If all pass: "All {N} test cases passed across {M} spec suites."
- If failures exist:
  - "Run `/speckit-runtests` to re-run a single spec's suite."
  - "Run `/run-full-suite 004` to isolate a specific spec."
- If any suites show `NO SCRIPTS`: "Run `/speckit-scripttest` on those features to generate scripts."
- Exit code: surface the failed count — if any `FAIL > 0`, treat the overall run as failed (use `exit 1` semantics when running in terminal).