qa-tester

The full skill.

Original instructions from the publisher’s SKILL.md

# QA Tester

All test types (CLI / Dashboard UI / MCP protocol) are **fully automated** — zero human intervention.

Optional modifiers: append section letter (`run QA G`) or test ID (`run QA G-01`).

---

> ## ⛔ IRON RULES
>
> ### Rule 1: STRICTLY SERIAL
> Pick ONE test → Run ONE command → Wait for output → Record ONE row in `.github/skills/qa-tester/QA_TEST_PROGRESS.md` → THEN pick next.
> NEVER run two tests in one command. NEVER record two rows in one file edit. NEVER use parallel tool calls. NEVER plan the next test before recording the current one.
>
> ### Rule 2: PASS = TERMINAL OUTPUT EVIDENCE
> ✅ means you ran a command **in THIS session** and the Note contains **concrete values** copied from that output.
> No terminal output for THIS test → mark ⬜. NEVER ✅.
>
> ### Rule 3: ZERO CROSS-REFERENCING
> NEVER write "Verified via X-YZ", "Already verified in C-02", "Same as…", "Similar to…".
> Even if G-05 tests the same flag as C-02, run G-05 independently and paste its own output.
>
> ### Rule 4: ZERO INFERENCE
> **BANNED Note patterns** (caught by validator):
> "Code uses…" / "Dataclass validates…" / "auto-creates…" → Reading code ≠ testing.
> "Should work…" / "Would raise…" / "Expected behavior…" → Speculation ≠ testing.
> "Parameter accepted" / "Config controls behavior" → Vague. No output = no pass.
> If you didn't run a command and see output for THIS test, mark ⬜.
>
> ### Rule 5: ADVERSARIAL MINDSET
> Find bugs, not confirmations. 10+ passes with zero bugs → re-examine your rigor.
>
> ### Rule 6: SECTION-END VALIDATION
> After completing a section, run `python .github/skills/qa-tester/scripts/qa_validate_notes.py`.
> Re-execute any flagged test before moving to next section.

---

## Pipeline (strictly serial)

```
1. Pick ONE pending test (by ID order)
2. Set system state if needed
3. Run ONE command — WAIT for output
4. Verify ALL assertions from Expected Result vs ACTUAL output
5. Fix if needed (≤3 rounds)
6. ⛔ GATE: Edit `.github/skills/qa-tester/QA_TEST_PROGRESS.md` (row + counters) — ONE row per edit
7. Only NOW return to step 1
```

> Activate `.venv` before any `python` command: `.\.venv\Scripts\Activate.ps1`

---

## Step 1: Pick Target

1. Read `.github/skills/qa-tester/QA_TEST_PLAN.md` for test steps and expected results.
2. Read `.github/skills/qa-tester/QA_TEST_PROGRESS.md` for current status.
3. User-specified section/ID → scope to that. Otherwise → first ⬜ pending test.
4. If any 🔧 tests exist, re-test those first.
5. Execute in section order (A→O), within section in ID order.

### Test Categories

| Sections | Type | Execution Method |
|----------|------|-----------------|
| A–F | Dashboard UI | AppTest headless — see [references/test_patterns.md](references/test_patterns.md) |
| G, H, I | CLI | Terminal commands, check exit code + stdout |
| J | MCP Protocol | JSON-RPC subprocess — see [references/test_patterns.md](references/test_patterns.md) |
| K, L | Provider Switch | `qa_config.py apply <profile>` → run CLI/Dashboard |
| M | Config & Fault Tolerance | Modify settings → run CLI → verify error handling |
| N, O | Data Lifecycle | Use `qa_multistep.py <TEST_ID>` |

---

## Step 2: Set System State

```
Empty          → python .github/skills/qa-tester/scripts/qa_bootstrap.py clear
Baseline       → python .github/skills/qa-tester/scripts/qa_bootstrap.py baseline
DeepSeek       → python .github/skills/qa-tester/scripts/qa_config.py apply deepseek
Rerank_LLM     → python .github/skills/qa-tester/scripts/qa_config.py apply rerank_llm
NoVision       → python .github/skills/qa-tester/scripts/qa_config.py apply no_vision
InvalidKey     → python .github/skills/qa-tester/scripts/qa_config.py apply invalid_llm_key
InvalidEmbedKey→ python .github/skills/qa-tester/scripts/qa_config.py apply invalid_embed_key
Any            → no state change needed
```

After config-profile tests → `python .github/skills/qa-tester/scripts/qa_config.py restore`
Check state → `python .github/skills/qa-tester/scripts/qa_bootstrap.py status`

---

## Step 3: Execute & Verify

### CLI Tests (G, H, I, parts of K/L/M)

1. Read the test's **Steps** column from `.github/skills/qa-tester/QA_TEST_PLAN.md`.
2. Run the exact command in terminal.
3. Compare output against **Expected Result**:
   - **Ingest**: exit=0, output has stage names (load/split/transform/embed/upsert)
   - **Query**: exit=0, results have source_file and score
   - **Error**: exit≠0, stderr has descriptive error (not raw stack trace)
   - **Idempotency**: second run shows "skipped", no duplicates

### Dashboard Tests (A–F)

Read [references/test_patterns.md](references/test_patterns.md) for AppTest templates, interaction patterns, and file upload workarounds.

Key points:
- Write one inline Python script per test, render via `AppTest.from_function`
- Print specific element values (metric labels, selectbox options, expander labels)
- For file uploads: ingest via CLI first, verify via AppTest
- For data mutations: call `DataService` directly, verify via AppTest/CLI

#### ⚠️ AppTest `st.tabs()` Limitation — MUST READ before testing any tab content

AppTest renders **all tab contents simultaneously** into a flat element list. There is **no spatial isolation** between tabs — you cannot determine which element belongs to which tab from `at.info`, `at.metric`, `at.markdown` etc.

**Consequence:** If a test step says "click tab X → see content Y inside tab X", AppTest **cannot confirm the spatial relationship**. Even if content Y exists in the output, it may come from a different region (e.g., a diagnostics area above the tabs, or a different tab).

**Mandatory check for ALL tab-content tests:**

1. **First verify the tab label exists:** Print `[t.label for t in at.tabs]` (if `at.tabs` is available) OR check the page source for the tab label string. If the expected tab label is **absent** → mark ❌ immediately — the tab does not exist.
2. **Then verify the content element exists** in the AppTest output.
3. **Acknowledge the limitation in the Note:** Append `[AppTest: tab isolation unverifiable]` to the Note.

**Example — tab does NOT exist:**
```python
# For a test expecting "🟪 Rerank" tab with "skipped" message:
tab_labels = [el.value for el in at.markdown if '🟪' in el.value or 'Rerank' in el.value]
print('Rerank tab label found:', tab_labels)
# If empty → tab absent → mark ❌ with note "Rerank tab not rendered (stage absent)"
```

**BANNED pattern:** Finding a content element (e.g., "Rerank skipped" info) and concluding the tab test PASSED — the element may be from a diagnostics region, not from inside the tab.

### MCP Tests (J)

Read [references/test_patterns.md](references/test_patterns.md) for JSON-RPC script template and assertion matrix.

Primary method: `pytest tests/e2e/test_mcp_client.py -v` covers most J-* cases.

### Multi-Step Tests (N, O, M config, L-07)

Tests with 3+ sequential steps **MUST** use the runner script:

```
python .github/skills/qa-tester/scripts/qa_multistep.py <TEST_ID>
```

**Supported:** `N-01`, `N-03`, `N-04`, `N-05`, `N-06`, `O-07`, `M-03`, `M-04`, `M-05`, `M-06`, `M-10`, `M-11`, `L-07`

The script executes every sub-step, prints ACTUAL values at each step, outputs `VERDICT: PASS/FAIL`. Copy the VERDICT and key step values into the Note.

For tests NOT in the script, run commands from `.github/skills/qa-tester/QA_TEST_PLAN.md` manually and paste output.

---

## Step 4: Fix & Retry (≤3 rounds)

1. **Diagnose**: code bug / config issue / missing data / test plan error?
2. **Fix**: minimal change only. Record file/line in Note.
3. **Retry**: re-run same command.
4. After 3 failed rounds → mark ❌ with detailed notes.
5. If fix touches shared code → re-run previously-passed tests in same section.

---

## Step 5: Record Results

**⛔ GATE — do this BEFORE picking the next test.**

Edit `.github/skills/qa-tester/QA_TEST_PROGRESS.md`: update ONE test row + summary counters. ONE row per file edit.

### ✅ PASS Requirements

All must be true:
1. Ran the command **in this session**
2. Observed actual output **from that command**
3. Verified **every** assertion in Expected Result
4. Note contains **≥2 concrete values** from terminal output

### Note Format

```
<method>: <value_1>, <value_2>[, ...]
```

- **CLI**: `exit=0, stdout: 'Total chunks: 3', source_file=simple.pdf`
- **AppTest**: `at.metric[0].label='Total traces', at.metric[0].value=6`
- **Multi-step**: `Step1: exit=0, chunks=3. Step2: sources=[simple.pdf]. Step3: deleted=1. Step4: sources=[]`
- **Bad** (BANNED): `"Already verified in C-02"`, `"Code uses yaml.safe_load"`, `"Should work because..."`, `"Parameter accepted"`

### Status Icons

| Icon | Meaning |
|------|---------|
| ✅ | Pass — all assertions verified against actual output |
| ❌ | Fail — failed after 3 fix attempts |
| ⏭️ | Skip — missing third-party API key (K-series only) |
| 🔧 | Fix applied — needs re-test |
| ⬜ | Pending — not yet tested |

### Counters

Update in the same edit: `✅ Pass: X | ❌ Fail: Y | ⏭️ Skip: Z | 🔧 Fix: W | ⬜ Pending: P` (must sum to Total).

### Section-End Gate

After each completed section:
```
python .github/skills/qa-tester/scripts/qa_validate_notes.py
```
Re-execute any flagged test. Do NOT proceed until 0 flags.

---

## Key Paths

| File | Purpose |
|------|---------|
| `.github/skills/qa-tester/QA_TEST_PLAN.md` | Test steps and expected results |
| `.github/skills/qa-tester/QA_TEST_PROGRESS.md` | Execution status and notes |
| `config/settings.yaml` | System configuration |
| `scripts/ingest.py` / `query.py` / `evaluate.py` | CLI commands |
| `tests/e2e/test_mcp_client.py` | MCP E2E tests |
| `tests/e2e/test_dashboard_smoke.py` | Dashboard smoke tests |
| `tests/fixtures/sample_documents/` | Test PDFs |
| `tests/fixtures/golden_test_set.json` | Evaluation golden set |

## Test Documents

| File | Language | Pages | Images |
|------|----------|-------|--------|
| `simple.pdf` | EN | 1 | 0 |
| `with_images.pdf` | EN | 1 | 1 |
| `complex_technical_doc.pdf` | EN | ~8 | 3 |
| `chinese_technical_doc.pdf` | ZH | ~8 | 0 |
| `chinese_table_chart_doc.pdf` | ZH | ~6 | 3 |
| `chinese_long_doc.pdf` | ZH | 30+ | 0 |
| `blogger_intro.pdf` | ZH | ~4 | 2 |

All in `tests/fixtures/sample_documents/`.