cx-ai-center

The full skill.

Original instructions from the publisher’s SKILL.md

# AI Center Skill

**This is the tool for anything about AI/GenAI applications** — both **questions** about their
behavior (prompts/responses, quality, hallucinations, guardrails, security, cost/tokens, errors,
latency — everything AI apps expose through their GenAI spans/tags) and **actions** to manage them
(applications, evaluations/policies, policy↔app links, model pricing). If a request touches an AI
application or its GenAI telemetry, use this skill.

Coralogix **AI Center** observes, evaluates, and guards GenAI/LLM applications. This skill
answers questions about AI apps from two sources:

- **Configuration** (this skill's `cx ai-center` commands): the AI application inventory,
  configured evaluations/policies, coverage, custom evaluations, and model pricing — none of
  which live in span telemetry.
- **Telemetry** (GenAI spans): what users asked, how the model answered, cost, tokens,
  latency, errors, tool calls, and eval/guardrail verdicts — queried with **`cx spans '<DataPrime>'`**.
  See [references/ai-center-queries.md](references/ai-center-queries.md) for the full,
  runnable query library, span schema, and playbooks.

Match the source to the question: *"which apps lack guardrails"* → config
(`cx ai-center applications list`); *"what are users asking my chatbot"* → telemetry
(`cx spans '…'`, reading the conversation from the GenAI spans). Some questions need **both** —
e.g. *"is my chatbot's PII policy actually catching PII?"* joins config (is the policy enabled)
with telemetry (the PII verdicts + the messages).

---

## Destructive Operation Safety

All write operations (`create`, `update`, `delete`, `add`, `remove`, `set`)
require interactive confirmation. `ai-center` is a **risky** command, so writes are also
gated by `allow_risky_commands` in `~/.cx/config.toml`. To skip the prompt in scripts, pass
`--yes`.

**IMPORTANT: NEVER pass `--yes` without explicit user approval.** Before executing any write:
1. Describe the exact operation to the user (what will be created/modified/deleted/linked).
2. Wait for the user to confirm.
3. Only then execute with `--yes`.

Read operations (`list`, `get`, `coverage`, `list-for-application`, `model-pricing get`) do
not require confirmation and can be run freely.

### Read-Only Mode
Use `--read-only` (or `CX_READ_ONLY=1`) to block every write at the CLI level — safe for
exploration.

### Agent Mode
When running inside an AI agent (Claude Code, Cursor, Codex, …), cx detects it and — instead of
showing a confirmation prompt that would hang forever (no human is there to type y/n) — stops
immediately with an error telling you to get the user's approval, then re-run with `--yes`.

### No delete commands (by design)
The CLI intentionally exposes **no delete** for custom-evaluation policies, AI applications, or
model pricing — even though the AI v3 API has those delete endpoints, `cx ai-center` does not
surface them.
- **Custom-evaluation policy:** can't be deleted; to take it off an app, detach with
  `custom-evaluations remove` (the policy object survives and can be re-attached).
- **Model pricing:** no delete command. It's **team-wide** (not per-app), so to change or clear
  it, run `model-pricing set` with a new map (an empty map `{}` clears all overrides) — `set`
  replaces the whole set.

---

## Golden rule

For **content** questions (quality, hallucination, sentiment, topics) read the actual
conversation and cite the `traceID` — don't rely on verdict tags alone. The transcript lives in
one of two conventions (`gen_ai.input.messages`/`output.messages`, or the older indexed
`gen_ai.prompt.<n>`/`completion.<n>` tags); read it with the **Reading conversations (content
questions)** queries in the library, which handle both and exclude the system prompt and tool
traffic. Full guidance:
[references/ai-center-queries.md](references/ai-center-queries.md).

---

## CLI Commands

**Show names to the user; use UUIDs only internally.** When presenting results, refer to apps
and evaluations by their human names (application/subsystem, evaluation name), not raw UUIDs.
The UUID is only needed to *call* a by-id or write command — resolve it yourself from the
matching `list` command (never guess or make the user paste a UUID).

### Applications (inventory + guarded status)

| Command | Purpose |
|---------|---------|
| `cx ai-center applications list` | List AI apps incl. `guardrailsIntegrated` (guarded) status |
| `cx ai-center applications list --evaluation-type <TYPE>` | Filter to apps using an eval type (repeatable) |
| `cx ai-center applications list --page-size <N> --page-offset <N>` | Paginate |
| `cx ai-center applications get <application-id>` | One application by UUID |

### Evaluations (configured policies on apps)

| Command | Purpose |
|---------|---------|
| `cx ai-center evaluations list` | All configured evaluations |
| `cx ai-center evaluations list --application <app> --subsystem <sub>` | Scope to one app (the pair) |
| `cx ai-center evaluations list --evaluation-type <TYPE>` | Filter by type — `<TYPE>` is the API enum (e.g. `PII`, `TOXICITY`, `PROMPT_INJECTION`; the keys from `coverage`), **not** the lowercase form |
| `cx ai-center evaluations get <evaluation-id>` | One evaluation by UUID |
| `cx ai-center evaluations create --from-file eval.json` | Create/enable an evaluation *(write)* |
| `cx ai-center evaluations update <evaluation-id> --from-file patch.json` | Partial update *(write)* |
| `cx ai-center evaluations delete <evaluation-id>` | Remove an evaluation from its app *(write)* |

### Custom evaluations (policies) & application links

| Command | Purpose |
|---------|---------|
| `cx ai-center custom-evaluations list` | All custom evaluation policies |
| `cx ai-center custom-evaluations list-for-application <application-id>` | Policies linked to one app |
| `cx ai-center custom-evaluations create --from-file policy.json` | Create a custom policy *(write)* |
| `cx ai-center custom-evaluations update <id> --from-file patch.json` | Partial update *(write)* |
| `cx ai-center custom-evaluations add <evaluation-id> <application-id>` | Attach a policy to an app *(write)* |
| `cx ai-center custom-evaluations remove <evaluation-id> <application-id>` | Detach (reversible) *(write)* |

> **By-id is prebuilt-only.** `evaluations get <id>` fetches a **prebuilt/configured** evaluation.
> Custom policies have **no** get-by-id — find one via `custom-evaluations list` /
> `list-for-application` and match by `id`/name.

### Coverage & model pricing

| Command | Purpose |
|---------|---------|
| `cx ai-center coverage` | Map of each evaluation type → number of apps using it (coverage / gap analysis) |
| `cx ai-center model-pricing get` | Team's custom per-model pricing overrides |
| `cx ai-center model-pricing set --from-file prices.json` | Set team pricing (team-wide, new data only) *(write)* |

The `--from-file` bodies for `evaluations` and `custom-evaluations` match the AI v3 API
shape verbatim; use `-` to read JSON from stdin. For `evaluations create`, `target` is
**required** and must be uppercase (`PROMPT` or `RESPONSE`); for `custom-evaluations create`,
`name`, `instructions`, and `policyType` are required. **Exception:** `model-pricing set` takes just
the raw `model→price` map — cx wraps it as `{"prices": …}` for you, so do **not** include the
outer `prices` envelope. Each model maps to a price object; all four fields are optional doubles
(USD per **one million** tokens), omit the ones that don't apply:

```json
{
  "gpt-4o": {
    "inputPricePerMillionTokens": 2.5,
    "outputPricePerMillionTokens": 10,
    "cacheReadPricePerMillionTokens": 1.25,
    "cacheWritePricePerMillionTokens": 3.75
  }
}
```

An empty map `{}` clears all overrides (set replaces the whole set — it's team-wide, new data only).
`model-pricing get` returns the wrapper `{ "pricing": { "id", "companyId", "prices": { … } } }` — the
per-model overrides live under `prices` (empty when none are set).

---

## Common workflows

### Inventory & guardrail gaps
```bash
# Which apps are NOT guarded?
cx ai-center applications list -o json | jq '[.[] | select(.guardrailsIntegrated==false)]'
```

### Enable a policy on an app (write — confirm first!)
```bash
# 1. Describe to the user; 2. get approval; 3. then:
cx ai-center evaluations create --from-file eval.json --yes
# eval.json: { "application": "...", "subsystem": "...", "target": "PROMPT", "config": { "<type>": {...} }, "isEnabled": true }
# `target` is REQUIRED and must be UPPERCASE — "PROMPT" or "RESPONSE" (the API rejects lowercase / a missing target).
```

### Read the actual conversations (telemetry, not config)
Use `cx spans` with the query library in
[references/ai-center-queries.md](references/ai-center-queries.md) — reading messages, cost,
latency, errors, tool calls, and per-user analysis.

---

## Key principles

- **Config vs. telemetry:** inventory / evaluations / policies / coverage / pricing → `cx ai-center`;
  content / cost / latency / errors / verdicts → GenAI spans via `cx spans`. Don't answer one
  from the other.
- **Confirm before writes.** Describe the operation, get approval, then run with `--yes`.

---

## Related Skills

- `cx-telemetry-querying` — general logs/spans/metrics/DataPrime querying (the engine behind
  the `cx spans` queries used here).
- `cx-olly` — the conversational AI assistant (`cx olly ask`).