upstream-mlx-vlm-issues

The full skill.

Original instructions from the publisher’s SKILL.md

# Upstream mlx-vlm Issue Drafts

Turn a failure into concise, actionable **mlx-vlm** issue Markdown. Default
output is issue-ready text or an improved local draft under `src/output/issues/`.
File with `gh issue create`, or open a PR upstream, only when the user asks
for it: an issue is published under the user's name and cannot be taken back.

Adapted from the upstream `reproducible-github-issues` skill
([`skills/skills/reproducible-github-issues`](https://github.com/Blaizzy/mlx-vlm/tree/main/skills/skills/reproducible-github-issues),
added by [Blaizzy/mlx-vlm#1747](https://github.com/Blaizzy/mlx-vlm/pull/1747)), aligned
with this repo’s retained artifacts and pip/conda workflow.

## When to use which artifact

| Situation | Start here |
| --------- | ---------- |
| Hard crash already drafted | `src/output/issues/issue_*.md` |
| Aggregate run / observations | `src/output/reports/diagnostics.md` |
| Sweep overview | `src/output/issues/run_summary.md` |
| Exact machine facts | `src/output/results.jsonl` (metadata header + per-model rows) |
| Environment stamp | `src/output/environment.log`, report provenance blocks |
| Need a minimal native command first | `native-mlx-vlm-repro` skill |

Issue drafts are created only for **hard actionable crashes** by default.
Completed-but-unusable observations stay in diagnostics unless the user asks for
a separate upstream write-up.

## Required information

Collect and verify; mark unavailable facts as unknown rather than inferring them:

- mlx-vlm version **or** git commit; install method (PyPI, editable, branch).
- Python version, macOS version, chip (for example M-series), Metal vs CUDA.
- Exact model ID or local path and **resolved revision** when known.
- Model source: HF cache, local conversion, or custom checkpoint.
- Exact native CLI command and/or Python `load` → `apply_chat_template` →
  `generate` script (from the draft when available).
- For server-only surfaces: startup command **and** `curl` request body
  (see server section below). Prefer `curl` over client SDKs.
- Input media facts: image dimensions; whether the input can be shared.
- Expected vs actual behavior; full root exception + relevant traceback.
- Whether the failure reproduces **outside** `check_models` with native mlx-vlm.

## Repro minimization

1. Establish the original native failure with the recorded checkpoint revision,
   prompt, media and effective decoding settings, then reduce it one factor at
   a time to the smallest command or request that still fails. A greedy run is
   a separate control, not a replacement for a sampling-related reproduction.
2. Remove private paths, tokens, and unrelated environment variables.
3. Prefer public models and small/synthetic media when possible.
4. One image before multi-input; one model per process.
5. State clearly if it only fails with a private checkpoint.
6. Read the model family's `mlx_vlm/models/<family>/README.md` before filing
   (the draft's "Family README in installed mlx-vlm" fact names it). If it
   recommends a different checkpoint, an authorized test of that checkpoint
   can supply comparison evidence after reproducing the original failure.
   Record which checkpoint each outcome belongs to; do not imply permission
   for additional downloads or inference merely because an issue is requested.
7. Lead with **root exception + first frames inside mlx-vlm / transformers /
   model code**; keep long harness stacks under an optional details block.
8. Commands use `python -m …` under conda + pip, not `uv run`: the environment is conda-managed, and uv would install outside it.

## Establish whether the defect is in the checkpoint or runtime

mlx-vlm maintainers receive many reports that turn out to be checkpoint
problems. Every generated draft opens with "Before filing: rule out the
checkpoint", listing the checks the run's own evidence calls for (no loader
for the model type, processor files missing from the snapshot, a family
README, a load-phase failure). Work through them first:

- No loader for the model type: a support request, not a bug. Look for an
  existing request; do not file a crash report.
- An alternative checkpoint running cleanly establishes a difference, not
  fault ownership. A valid configuration can expose an upstream compatibility
  bug that another checkpoint avoids. Compare the relevant config, processor,
  weights and failing code path before attributing the defect.
- Route a demonstrated checkpoint defect to its Hugging Face repository and a
  demonstrated runtime defect to mlx-vlm. If ownership remains uncertain,
  describe the evidence and uncertainty rather than declaring a checkpoint
  faulty or suppressing a potentially valid upstream report. Publishing to
  either destination still needs the user's authorization.

## Maintainer quality bar

Ready when an mlx-vlm maintainer can run **one** native command (or one server
start + one `curl`) and see the same failure without asking for basic
environment or model details.

Prefer factual language:

- Record phase, package tag, revision, and exact error text.
- Do not invent semantic quality scores or blame narratives.
- Distinguish harness preflight failures from native generate failures.
- Neutral observations (declared EOS/thinking wrappers, unchanged draft fields)
  are reproduction facts, not automatic bug claims.
- A properly closed thinking block followed by substantive final text is neutral,
  including when the rendered prompt seeded its opening delimiter. File only the
  independently reproduced defect: an incomplete block, exhausted token budget,
  absent final answer, or undeclared control/role token.
- Do not use a sanitised gallery preview as the exact media input unless its
  digest matches the retained run manifest. With local-only media, include its
  characteristics and say that the exact input is not published.

## Issue template

Use this shape when authoring or rewriting paste-ready Markdown. Fill from
existing artifacts rather than paraphrasing away exact errors.

````markdown
### Summary

<One sentence describing the failure.>

### Environment

- MLX-VLM:
- mlx:
- Python:
- OS / macOS:
- Hardware / MLX device:
- Install method: <PyPI | editable path | branch>
- check_models (optional context): <version/revision if relevant>

### Model

- Model:
- Resolved revision:
- Source: <HF cache | local path | converted checkpoint>
- Trust remote code: <yes/no>
- Family README: <path, and the checkpoint it recommends, if any | none>

### Reproduction

```bash
python -m mlx_vlm.generate \
  --model <model> \
  --image <image> \
  --prompt '…' \
  --max-tokens 128 \
  --temperature 0.0
```

Optional Python (when CLI cannot express the failing kwargs):

```python
# load → apply_chat_template → generate; one model per process
```

### Expected behavior

<What should have happened.>

### Actual behavior

<What happened instead.>

### Logs / traceback

```text
<root exception and trimmed upstream frames>
```

### Inputs

<Shareable media description; dimensions; note if synthetic repro works.>
````

## Server-only failures

`check_models` benchmarks by looping over `mlx_vlm.generate.stream_generate`
(the same path upstream `generate()` wraps) and does **not** cover HTTP server
behavior. For `/v1/chat/completions`, `/v1/responses`, streaming,
structured outputs, tools, continuous batching, or `/v1/models` mismatches:

```bash
python -m mlx_vlm.server --model <model-or-path> --port 8080
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models
```

Capture startup command, server logs, request JSON, status code, and body.
Keep streaming and non-streaming repros separate. Still use conda + pip, not uv.

## Classification hints

| Bucket | Typical signals |
| ------ | --------------- |
| Model / config | missing `image_processor`, bad `config.json`, chat template unset |
| mlx-vlm runtime | failure inside generate / apply_chat_template / processor path |
| Harness preflight | check_models validator before native generate |
| Environment | Metal OOM, disk, permissions |
| Connectivity | download / hub disconnect — usually indeterminate, not a model crash |

## After drafting

- If the user only wanted local improvement: leave the Markdown in chat or under
  `src/output/issues/` without filing.
- If the user asks to file upstream: use their account/process; keep the body
  GitHub-sized; link or attach diagnostics only when needed.
- Do not treat observation-only gallery rows as automatic upstream bugs without
  a minimized native repro and expected/actual behavior.