Agent Evals

Weiser can evaluate natural-language-to-SQL / BI agents that sit on top of a semantic layer — question in, SQL + answer out. Like Weiser's own data quality checks, agent evals are configured declaratively in YAML (Evals-As-YAML): which tools the agent may call, which prompt and model it runs with, and the test cases themselves all live in the config file, not in code.

Agent evals are a second, complementary capability to data quality checks: checks validate that your data is trustworthy; evals validate that your agent answers questions about that data correctly.

Core Concepts

Concept	Config section	What it is
Semantic layer	semantic_layers	The schema the agent queries against — Cube.js or generic SQL introspection over any Weiser datasource
Agent variant	agent_variants	A declaratively-configured agent shape: framework, model, system prompt, tool set. Variants of the same agent differ by config, not by Python
Arm	eval_suites[].arms	A named pairing of an agent variant with a semantic layer — one point in a comparison
Eval suite	eval_suites	One or more arms compared against a shared set of goldens, scored by a set of metrics
Golden	eval_suites[].goldens / golden_set	A single test case: a natural-language question plus optional reference values, expected views, and metadata
Metric	eval_suites[].metrics	How each run is scored — deterministic checks and/or LLM-judge criteria

The typical experiment is an ablation: two arms that are identical except for one variable (a tool on/off, a prompt change, a model swap) compared against the same goldens. When a suite has exactly two arms, Weiser prints a warning if the arms differ in more than one dimension — comparing arms that change multiple variables at once makes the resulting delta unexplainable.

How a Run Works

For each arm, each golden, and each repeat:

1. The semantic layer adapter exposes a standard four-tool toolset — list_views, describe_view, query, submit_answer — filtered to the variant's tools list.
2. The agent (built by a per-framework factory function you write once) runs the question and produces a trace: tool calls, predicted SQL, query results, final answer, cost, and turn count.
3. Each configured metric scores the trace (0.0–1.0 per criterion).
4. Weiser's own DQ checks are wired in: the tables the agent's SQL touched are cross-referenced against your configured checks: (or their most recent stored results), and each failing row gets a failure attribution — clean, agent, data_quality, or judge_uncertain. A wrong answer caused by a known data incident is attributed to data quality, not to the agent.
5. Results are dual-written: a flat summary row per (golden, arm) into your existing metric store (visible in the Weiser dashboard with no changes), plus a full JSONL trace substrate under eval_results/ for comparison, gating, and failure analysis.

Typical Workflow

# 1. Validate the eval config without executing anything
weiser eval-compile evals.yaml --suite my_suite

# 2. Run the suite (train split while iterating)
weiser eval evals.yaml --suite my_suite --split train -v

# 3. Gauge score stability before trusting a delta
weiser eval evals.yaml --suite my_suite --split train --repeats 3

# 4. Generate more seed questions from the schema (review before committing)
weiser eval-synth evals.yaml --semantic-layer my_sl --out evals/synthetic.yaml

# 5. Gate a candidate run against a saved baseline in CI
weiser eval-gate --candidate candidate.jsonl --baseline baseline.jsonl --arm baseline

# 6. Verify an LLM-judge metric agrees with human labels
weiser eval-calibrate evals.yaml --suite my_suite --metric-name sql_soundness \
  --turns turns.jsonl --labels labels.jsonl

Results

• Scorecard: per-arm accuracy (overall / easy / hard), average turns, average cost, hit-limit rate, and a failure-attribution breakdown.
• Pairwise delta: with 2+ arms, the overall-accuracy delta of each arm versus the first arm.
• Variance report: with --repeats N, per-question mean/std across identical repeats — a question whose score swings widely is telling you something (prompt ambiguity, a genuinely hard case, or judge noise) before it shows up as a mysterious regression later.
• Trace substrate: eval_results/{suite}_{split}_{run_id}.jsonl — one JSON line per (golden, arm, repeat) containing the full result row and agent trace. This is what eval-gate reads.

Documentation

• Configuration — the agent_variants, semantic_layers, and eval_suites sections in detail
• Commands — weiser eval, eval-compile, eval-synth, eval-gate, eval-calibrate
• Metrics — deterministic metrics, the LLM judge, and ready-made judge presets