Agent Eval Commands

Weiser ships five commands for the agent eval workflow. All commands that read a config file support --env-file / -e (default .env) and --verbose / -v, the same as weiser run and weiser compile.

Command	Purpose
weiser eval	Run an eval suite
weiser eval-compile	Validate eval config without executing
weiser eval-synth	Generate synthetic seed questions from a semantic layer's schema
weiser eval-gate	CI regression gate comparing a candidate run against a baseline
weiser eval-calibrate	Verify an LLM-judge metric against human labels

weiser eval

Run an agent eval suite.

weiser eval <config.yaml> --suite <name> [options]

Option	Default	Description
--suite	required	Name of the eval_suites entry to run
--split	all	Filter goldens by split (train / held_out)
--dq-mode	latest	latest: read the most recent stored DQ check result. live: run the configured DQ checks now
--repeats	1	Run each golden this many times per arm and print a per-question variance report — use to gauge how stable a score is before trusting a single run's delta
-v, --verbose	off	Print parsed config to stdout
-e, --env-file	.env	Path to a custom .env file

# Train split, 3 repeats for a variance report
weiser eval evals.yaml --suite lookup_tool_ablation --split train --repeats 3 -v

Output: a scorecard table per arm (accuracy overall/easy/hard, average turns, average cost, hit-limit rate), a failure-attribution breakdown for any failing rows, a pairwise delta table when the suite has 2+ arms, and the path to the JSONL trace substrate (eval_results/{suite}_{split}_{run_id}.jsonl). Flat summary rows are also written to your metric store and show up in the Weiser dashboard.

weiser eval-compile

Validate the eval config sections (agent_variants / semantic_layers / eval_suites) and resolve their references without executing anything.

weiser eval-compile <config.yaml> [--suite <name>] [options]

Option	Default	Description
--suite	all	Validate a single eval_suites entry instead of all
-v, --verbose	off	Print parsed config to stdout
-e, --env-file	.env	Path to a custom .env file

weiser eval-compile evals.yaml --suite lookup_tool_ablation

Fails with a clear error on unknown agent_variant, semantic_layer, or datasource references and on suites with no goldens. Prints a warning (without blocking) when a two-arm suite's arms differ in more than one dimension — e.g. arms 'baseline' and 'no_describe_view' differ in: tools, model -- are you testing more than one variable at once?.

weiser eval-synth

Generate synthetic seed questions from a semantic layer's schema, optionally biased toward views with a currently-failing Weiser DQ check. Writes a golden YAML file (source: synthetic, split: train) directly usable as an eval_suite's golden_set.

weiser eval-synth <config.yaml> --semantic-layer <name> --out <path> [options]

Option	Default	Description
--semantic-layer	required	Name of the semantic_layers entry to introspect
--out	required	Path to write the generated golden YAML file
--n-per-view	3	Number of synthetic questions to generate per view
--views	all	Comma-separated view names to restrict generation to
--model	anthropic:claude-sonnet-5	Seed-writer model — deliberately separate from both the agent under test and any judge model, to avoid contaminating grading with generation
--use-dq-hints / --no-dq-hints	on	Bias generated questions toward views with a currently-failing Weiser DQ check
-v, --verbose	off	Print parsed config to stdout
-e, --env-file	.env	Path to a custom .env file

weiser eval-synth evals.yaml --semantic-layer local_sl --out evals/synthetic.yaml

Synthetic goldens never come with held_out splits or reference_values — both stay human-curated. Review the output before promoting anything to held_out or adding reference values.

weiser eval-gate

CI regression gate: compares a candidate run's JSONL trace substrate against a baseline's for one arm, and exits non-zero on regression.

weiser eval-gate --candidate <run.jsonl> --baseline <baseline.jsonl> --arm <name> [options]

Option	Default	Description
--candidate	required	Path to the candidate run's JSONL trace substrate
--baseline	required	Path to the baseline run's JSONL trace substrate
--arm	required	Arm name to compare (must exist in both files)
--max-regression	3.0	Max allowed accuracy drop, in percentage points
--exclude-confounded / --include-confounded	exclude	Exclude data_quality-attributed candidate rows from the regression delta — a live DQ incident shouldn't fail an unrelated agent PR

weiser eval-gate \
  --candidate eval_results/candidate.jsonl \
  --baseline eval_results/baseline.jsonl \
  --arm baseline \
  --max-regression 3.0

Fails (exit code 1) if accuracy drops more than --max-regression percentage points, if the hit-limit rate increases at all (zero tolerance for new hard failures), or if either file has no rows for the arm. Prints a JSON summary (passed, delta_pp, excluded_confounded_count, reasons, per-arm summaries) for CI logs.

weiser eval-calibrate

Score a calibration set with an llm_judge metric and report agreement with human labels (MAE + Spearman correlation). Exits non-zero if the correlation falls below --threshold — an uncalibrated judge should not silently pass.

weiser eval-calibrate <config.yaml> --suite <name> --metric-name <name> \
  --turns <turns.jsonl> --labels <labels.jsonl> [options]

Option	Default	Description
--suite	required	Name of the eval_suites entry the metric is defined in
--metric-name	required	Name of the llm_judge metric within that suite to calibrate
--turns	required	Path to the calibration turns JSONL file
--labels	required	Path to the human labels JSONL file
--threshold	0.5	Minimum acceptable Spearman correlation against human labels
--out	eval_results/calibration	Directory to write the calibration report JSON to
-v, --verbose	off	Print parsed config to stdout
-e, --env-file	.env	Path to a custom .env file

The turns file holds pre-recorded agent turns, one JSON object per line:

{"turn_id": "t1", "golden": {"id": "q1", "input": "..."}, "trace": {"question": "...", "tool_calls": [], "predicted_sqls": ["..."], "final_answer": "...", "query_results": [], "hit_limit": false, "elapsed_s": 1.0, "cost_usd": 0.0, "prompt_tokens": 0, "completion_tokens": 0}}

The labels file holds one row per (turn, criterion):

{"turn_id": "t1", "criterion": "sql_soundness", "human_score": 0.8, "human_rationale": "sound but suboptimal join", "labeled_by": "paco", "labeled_at": "2026-08-01"}

Typical source of turns: real held-out runs plus deliberately-injected edge cases. The report is written to --out/{judge_prompt_version}.json and includes the judge prompt version, so a rubric edit is never silently compared against results scored under the old rubric.