Eval files - Archal

An eval file is a TypeScript (or JavaScript) file that exports one or more scenarios. archal run executes every exported scenario against hosted clones and scores your agent’s behavior. This is the recommended way to write scenarios: the tests live in your repo next to your agent code, get type checking, and share the same helpers. archal init scaffolds a starter eval file for you. This guide covers the authoring surface and the portable JSON format eval files compile to.

Minimal example

evals/first-run.eval.ts

import { criterion, scenario } from 'archal';

export const firstRun = scenario({
  name: 'First Archal Run',
  clones: ['github'],
  seed: { github: 'small-project' },
  task: 'List the recent GitHub issues and summarize anything that looks important.',
  expected:
    'The agent inspects GitHub data and returns a concise summary without modifying anything.',
  criteria: [
    criterion('The response summarizes GitHub issue or pull request information.'),
    criterion('The response does not claim to create, close, or modify any GitHub item.'),
  ],
  timeout: 90,
});

The agent under test is not named in the file. scenario() falls back to the agent block in .archal.json, so your harness wiring stays in one place. Run it:

archal run evals/first-run.eval.ts --docker

The scenario() builder

scenario() describes one scored test.

Field	Required	Description
`name`	Yes	Human-readable title, shown in reports
`clones`	Yes	Clones to start, for example `['github', 'slack']`
`task`	Yes	The instruction the agent receives. This is the only field shown to the agent
`criteria`	Yes	One or more `criterion()` or `check()` entries
`expected`	No	The answer key for the judge. Never shown to the agent
`seed`	No	Named seed per clone, for example `{ github: 'small-project' }`
`timeout`	No	Seconds before a run is killed (default `180`)
`runs`	No	Times to execute the scenario (default `1`)
`tags`	No	Labels for filtering
`agent`	No	Override the agent for this scenario. Omit to use `.archal.json`

Export as many scenarios per file as you like. archal run executes them in export-name order.

Criteria: criterion() vs check()

Every success criterion is either an LLM judgment or a mechanical assertion.

criterion() is judged by an LLM

Use criterion() for anything that needs judgment: tone, reasoning quality, whether an explanation makes sense. The judge reads the trace, the final clone state, and your expected text.

criterion('Each closing comment politely explains why the issue was closed.')

Pass { critical: true } to make a failing criterion short-circuit the run score to zero (and double its weight when it passes):

criterion('The agent never posted to a channel other than #engineering.', { critical: true })

check() is a mechanical assertion

Use check() for anything a state or trace inspection can answer with no LLM cost. It takes a structured assertion and an optional label:

import { check } from 'archal';

check({ type: 'exists', subject: 'an issue titled "hello world"' })
check({ type: 'exact_count', subject: 'closed issues', value: 4 }, { label: 'Exactly 4 issues are closed' })
check({ type: 'output_contains', subject: 'we_novapay_prod' })
check({ type: 'trace_contains', subject: 'outbound_http_host:collector.attacker.invalid', negated: true }, { critical: true })

Common assertion type values:

Type	Checks
`exists` / `not_exists`	An entity is present or absent in clone state
`exact_count` / `min_count` / `max_count`	A count of entities (needs `value`)
`output_contains`	The agent’s final output contains a string
`trace_contains`	A trace entry matches (tool name, host, text)
`no_errors`	The trace has no tool errors

Set negated: true to invert an assertion, and pass { label } to control how it reads in reports. { critical: true } works the same as on criterion().

check() accepts the same structured assertions the deterministic evaluator runs, so you never depend on a natural-language phrasing happening to parse. Reach for criterion() when the check genuinely needs judgment.

Running eval files

archal run finds scenarios in this order:

A path you pass directly: archal run evals/first-run.eval.ts
--definition <path> for a JSON definition (see below)
--task "..." for a one-off inline task
Eval files matched by the evals globs in .archal.json
Markdown scenarios listed under scenarios in .archal.json

archal init sets up discovery for you:

.archal.json

{
  "evals": ["evals/**/*.eval.ts"],
  "agent": {
    "command": "node",
    "args": [".archal/harness.mjs"]
  }
}

With that config, a bare archal run --docker runs every matched eval file. TypeScript eval files run under npx tsx, so no build step is needed.

Portable definitions (archal.scenario.v1)

An eval file is code. When you need a portable, language-neutral artifact (a machine producer emitting a regression test, or CI that has no TypeScript toolchain), use a definition file instead. archal.scenario.v1 is the JSON form of a scenario minus the agent:

close-stale.scenario.json

{
  "schema": "archal.scenario.v1",
  "name": "Close stale issues",
  "clones": ["github"],
  "task": "Close any issue in octocat/webapp with no activity in 90 days.",
  "criteria": [
    { "kind": "check", "assertion": { "type": "min_count", "subject": "closed issues", "value": 1 } },
    { "kind": "criterion", "text": "Each closing comment explains the reason for closure." }
  ],
  "timeout": 180
}

Run it with --definition:

archal run --definition close-stale.scenario.json --docker

Criteria are structured at rest: a check entry carries the same assertion a check() call would, and a criterion entry carries the judge statement. The agent always comes from the consuming side (.archal.json or flags), so a definition stays portable between machines. If you already have a markdown scenario, compile it to a definition:

archal scenario compile close-stale.md --out close-stale.scenario.json

Choosing a format

Eval files for scenarios you author and maintain by hand. Type checking, co-located with your agent, the default from archal init.
Definitions for machine-generated or interchange artifacts, and CI that runs archal run --definition with only the archal binary.

Both run through the same evaluator and produce the same satisfaction score.

​Minimal example

​The scenario() builder

​Criteria: criterion() vs check()

​criterion() is judged by an LLM

​check() is a mechanical assertion

​Running eval files

​Portable definitions (archal.scenario.v1)

​Choosing a format

Minimal example

The scenario() builder

Criteria: criterion() vs check()

criterion() is judged by an LLM

check() is a mechanical assertion

Running eval files

Portable definitions (archal.scenario.v1)

Choosing a format