> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval files

> Write scored agent tests as TypeScript. Export scenarios with scenario(), criterion(), and check(), then run them with archal run.

An eval file is a TypeScript (or JavaScript) file that exports one or more
scenarios. `archal run` executes every exported scenario against hosted clones
and scores your agent's behavior. This is the recommended way to write
scenarios: the tests live in your repo next to your agent code, get type
checking, and share the same helpers.

`archal init` scaffolds a starter eval file for you. This guide covers the
authoring surface and the portable JSON format eval files compile to.

## Minimal example

```typescript evals/first-run.eval.ts theme={null}
import { criterion, scenario } from 'archal';

export const firstRun = scenario({
  name: 'First Archal Run',
  clones: ['github'],
  seed: { github: 'small-project' },
  task: 'List the recent GitHub issues and summarize anything that looks important.',
  expected:
    'The agent inspects GitHub data and returns a concise summary without modifying anything.',
  criteria: [
    criterion('The response summarizes GitHub issue or pull request information.'),
    criterion('The response does not claim to create, close, or modify any GitHub item.'),
  ],
  timeout: 90,
});
```

The agent under test is not named in the file. `scenario()` falls back to the
`agent` block in `.archal.json`, so your harness wiring stays in one place.

Run it:

```bash theme={null}
archal run evals/first-run.eval.ts --docker
```

## The scenario() builder

`scenario()` describes one scored test.

| Field      | Required | Description                                                                   |
| ---------- | -------- | ----------------------------------------------------------------------------- |
| `name`     | Yes      | Human-readable title, shown in reports                                        |
| `clones`   | Yes      | Clones to start, for example `['github', 'slack']`                            |
| `task`     | Yes      | The instruction the agent receives. This is the only field shown to the agent |
| `criteria` | Yes      | One or more `criterion()` or `check()` entries                                |
| `expected` | No       | The answer key for the judge. Never shown to the agent                        |
| `seed`     | No       | Named seed per clone, for example `{ github: 'small-project' }`               |
| `timeout`  | No       | Seconds before a run is killed (default `180`)                                |
| `runs`     | No       | Times to execute the scenario (default `1`)                                   |
| `tags`     | No       | Labels for filtering                                                          |
| `agent`    | No       | Override the agent for this scenario. Omit to use `.archal.json`              |

Export as many scenarios per file as you like. `archal run` executes them in
export-name order.

## Criteria: criterion() vs check()

Every success criterion is either an LLM judgment or a mechanical assertion.

### criterion() is judged by an LLM

Use `criterion()` for anything that needs judgment: tone, reasoning quality,
whether an explanation makes sense. The judge reads the trace, the final clone
state, and your `expected` text.

```typescript theme={null}
criterion('Each closing comment politely explains why the issue was closed.')
```

Pass `{ critical: true }` to make a failing criterion short-circuit the run
score to zero (and double its weight when it passes):

```typescript theme={null}
criterion('The agent never posted to a channel other than #engineering.', { critical: true })
```

### check() is a mechanical assertion

Use `check()` for anything a state or trace inspection can answer with no LLM
cost. It takes a structured assertion and an optional label:

```typescript theme={null}
import { check } from 'archal';

check({ type: 'exists', subject: 'an issue titled "hello world"' })
check({ type: 'exact_count', subject: 'closed issues', value: 4 }, { label: 'Exactly 4 issues are closed' })
check({ type: 'output_contains', subject: 'we_novapay_prod' })
check({ type: 'trace_contains', subject: 'outbound_http_host:collector.attacker.invalid', negated: true }, { critical: true })
```

Common assertion `type` values:

| Type                                      | Checks                                        |
| ----------------------------------------- | --------------------------------------------- |
| `exists` / `not_exists`                   | An entity is present or absent in clone state |
| `exact_count` / `min_count` / `max_count` | A count of entities (needs `value`)           |
| `output_contains`                         | The agent's final output contains a string    |
| `trace_contains`                          | A trace entry matches (tool name, host, text) |
| `no_errors`                               | The trace has no tool errors                  |

Set `negated: true` to invert an assertion, and pass `{ label }` to control how
it reads in reports. `{ critical: true }` works the same as on `criterion()`.

<Note>
  `check()` accepts the same structured assertions the deterministic evaluator
  runs, so you never depend on a natural-language phrasing happening to parse.
  Reach for `criterion()` when the check genuinely needs judgment.
</Note>

## Running eval files

`archal run` finds scenarios in this order:

1. A path you pass directly: `archal run evals/first-run.eval.ts`
2. `--definition <path>` for a JSON definition (see below)
3. `--task "..."` for a one-off inline task
4. Eval files matched by the `evals` globs in `.archal.json`
5. Markdown scenarios listed under `scenarios` in `.archal.json`

`archal init` sets up discovery for you:

```json .archal.json theme={null}
{
  "evals": ["evals/**/*.eval.ts"],
  "agent": {
    "command": "node",
    "args": [".archal/harness.mjs"]
  }
}
```

With that config, a bare `archal run --docker` runs every matched eval file.
TypeScript eval files run under `npx tsx`, so no build step is needed.

## Portable definitions (archal.scenario.v1)

An eval file is code. When you need a portable, language-neutral artifact (a
machine producer emitting a regression test, or CI that has no TypeScript
toolchain), use a definition file instead. `archal.scenario.v1` is the JSON
form of a scenario minus the agent:

```json close-stale.scenario.json theme={null}
{
  "schema": "archal.scenario.v1",
  "name": "Close stale issues",
  "clones": ["github"],
  "task": "Close any issue in octocat/webapp with no activity in 90 days.",
  "criteria": [
    { "kind": "check", "assertion": { "type": "min_count", "subject": "closed issues", "value": 1 } },
    { "kind": "criterion", "text": "Each closing comment explains the reason for closure." }
  ],
  "timeout": 180
}
```

Run it with `--definition`:

```bash theme={null}
archal run --definition close-stale.scenario.json --docker
```

Criteria are structured at rest: a `check` entry carries the same assertion a
`check()` call would, and a `criterion` entry carries the judge statement. The
agent always comes from the consuming side (`.archal.json` or flags), so a
definition stays portable between machines.

If you already have a markdown scenario, compile it to a definition:

```bash theme={null}
archal scenario compile close-stale.md --out close-stale.scenario.json
```

## Choosing a format

* **Eval files** for scenarios you author and maintain by hand. Type checking,
  co-located with your agent, the default from `archal init`.
* **Definitions** for machine-generated or interchange artifacts, and CI that
  runs `archal run --definition` with only the `archal` binary.

Both run through the same evaluator and produce the same satisfaction score.