Agent Eval Configuration

Agent evals are configured in the same Weiser YAML file as your data quality checks, using three new optional top-level sections: agent_variants, semantic_layers, and eval_suites. All existing sections (datasources, checks, connections, includes, slack_url) work unchanged, and environment variable templating ({{VARIABLE_NAME}}) applies to the new sections too.

version: 1

datasources:
  - name: local_db
    type: postgresql
    uri: duckdb:///examples/eval_example.duckdb

checks:
  - name: merchants_not_empty
    dataset: merchants
    datasource: local_db
    type: row_count
    condition: gt
    threshold: 0

connections:
  - name: metricstore
    type: metricstore
    db_type: duckdb
    db_name: eval_example_metricstore.db

semantic_layers:
  - name: local_sl
    type: generic_sql
    datasource: local_db

agent_variants:
  - name: baseline
    framework: pydantic_ai
    entrypoint: examples.eval_agents.build_bi_agent
    system_prompt: "You are a careful BI analyst. Always verify column names before querying."
    tools: [list_views, describe_view, query, submit_answer]

  - name: no_describe_view
    framework: pydantic_ai
    entrypoint: examples.eval_agents.build_bi_agent
    system_prompt: "You are a careful BI analyst. Always verify column names before querying."
    tools: [list_views, query, submit_answer]

eval_suites:
  - name: lookup_tool_ablation
    arms:
      - name: baseline
        agent_variant: baseline
        semantic_layer: local_sl
      - name: no_describe_view
        agent_variant: no_describe_view
        semantic_layer: local_sl
    goldens:
      - id: merchant_count
        input: "How many merchants are there in total?"
        split: train
        level: easy
        reference_values:
          - metric: cnt
            expected: 3
            tolerance_pct: 0
        expected_views: [merchants]
    metrics:
      - type: schema_membership
        threshold: 1.0
      - type: reference_value_match
      - type: llm_judge
        name: sql_soundness
        threshold: 0.5
        params:
          evaluation_params: [input, predicted_sqls]
          criteria: >
            Given the question, is the submitted SQL structurally sound and a
            plausible way to answer it? Do not penalize style choices.

`semantic_layers`

A semantic layer tells the harness which schema the agent may query against. It references an existing datasource by name and reuses its connection details.

Parameter	Required	Description
`name`	Yes	Unique identifier, referenced by `eval_suites[].arms[].semantic_layer`
`type`	Yes	`cube` or `generic_sql`
`datasource`	Yes	Name of an existing `datasources` entry
`meta_api_url`	No	Cube.js only — base URL of the Cube API (e.g. `http://localhost:4000`)
`meta_api_token`	No	Cube.js only — API token; use `{{CUBE_API_TOKEN}}` for secrets

Generic SQL

Introspects tables/views directly from any Weiser SQL datasource via SQLAlchemy. No extra parameters needed:

semantic_layers:
  - name: local_sl
    type: generic_sql
    datasource: local_db

Cube.js

Reads measures/dimensions from the Cube REST meta API; queries still execute through the datasource connection:

semantic_layers:
  - name: cube_prod
    type: cube
    datasource: cube_prod_ds
    meta_api_url: "{{CUBE_META_URL}}"
    meta_api_token: "{{CUBE_API_TOKEN}}"

`agent_variants`

An agent variant is a declaratively-configured agent shape. entrypoint is a dotted path to a factory function you write once per agent family; every other field is a knob the harness passes into that factory, so variants of the same agent are pure config.

Parameter	Required	Default	Description
`name`	Yes	—	Unique identifier, referenced by `eval_suites[].arms[].agent_variant`
`framework`	Yes	—	Agent framework; currently `pydantic_ai`
`entrypoint`	Yes	—	Dotted path to a factory function, e.g. `examples.eval_agents.build_bi_agent`
`model`	No	framework default	Model string, e.g. `anthropic:claude-sonnet-5`
`system_prompt`	No	—	Inline prompt text or a path to a prompt file (Jinja2-rendered)
`tools`	No	all tools	Subset of `list_views`, `describe_view`, `query`, `submit_answer` to expose
`model_settings`	No	—	Passed through verbatim to the framework (e.g. `temperature`)
`max_turns`	No	`40`	Turn budget per question; hitting it is recorded as a first-class failure category
`extra`	No	—	Framework-specific escape hatch for anything not modeled above

The entrypoint factory

For pydantic_ai, the entrypoint is a function that takes the variant and the (already-filtered) toolset and returns a built Agent. Write one per agent family and reuse it across every variant:

# examples/eval_agents.py
from pydantic_ai import Agent


def build_bi_agent(variant, tools) -> Agent:
    return Agent(
        model=variant.model or "anthropic:claude-sonnet-5",
        system_prompt=variant.system_prompt or "You are a careful BI analyst.",
        tools=tools,
        model_settings=variant.model_settings,
    )

The tools handed to the factory are already filtered to variant.tools by the harness — that is what makes "compare an agent with/without a tool" a pure-YAML experiment. For a single-arm baseline of "whatever is actually in prod right now", the factory can ignore the knobs and return an already-built agent; extra carries anything framework-specific.

Ablation example

The canonical use case — add a tool, compare with/without. Same factory, same prompt, only the tool set differs:

agent_variants:
  - name: baseline
    framework: pydantic_ai
    entrypoint: myapp.eval_agents.build_bi_agent
    model: anthropic:claude-sonnet-5
    system_prompt: prompts/bi_system.md
    tools: [list_views, describe_view, query, submit_answer]

  - name: with_lookup_tool
    framework: pydantic_ai
    entrypoint: myapp.eval_agents.build_bi_agent
    model: anthropic:claude-sonnet-5
    system_prompt: prompts/bi_system.md
    tools: [list_views, describe_view, query, submit_answer, lookup_customer]

`eval_suites`

An eval suite pairs one or more arms against a shared set of goldens, scored by a set of metrics.

Parameter	Required	Description
`name`	Yes	Unique suite name, passed to `weiser eval --suite`
`arms`	Yes	One or more `EvalArm` entries; 2+ arms enables the pairwise comparison report
`golden_set`	No*	Path to an external golden file (`.yaml`, `.yml`, or `.jsonl`)
`goldens`	No*	Inline list of goldens; merged with `golden_set` if both are given
`metrics`	Yes	List of metric configurations (see Metrics)
`dq_scope`	No	Explicit dataset names to use for data-quality attribution; default is auto (derived from the tables the agent's SQL touched)

* At least one of golden_set or goldens must provide goldens — weiser eval-compile fails otherwise.

Arms

An arm is a named pairing of an agent variant with a semantic layer:

arms:
  - name: baseline
    agent_variant: baseline
    semantic_layer: local_sl
  - name: no_describe_view
    agent_variant: no_describe_view
    semantic_layer: local_sl

agent_variant and semantic_layer reference entries by name; weiser eval-compile resolves all references and fails on unknown names.

Goldens

A golden is a single test case. It can be declared inline in the suite or loaded from an external file via golden_set. External YAML files are {goldens: [...]}-shaped and go through the normal Weiser config loader (Jinja2 templating and includes: work); .jsonl files hold one golden dict per line for bulk, synthetic, or production-mined sets.

Parameter	Required	Default	Description
`id`	Yes	—	Unique identifier within the suite
`input`	Yes	—	The natural-language question
`split`	No	`train`	`train` or `held_out`; use `--split` to filter a run
`level`	No	`easy`	`easy` or `hard`; the scorecard reports accuracy per level
`source`	No	`hand_written`	`hand_written`, `synthetic`, or `production`
`reference_values`	No	—	Pinned expected values (see below); enables the `reference_value_match` metric
`reference_answer_text`	No	—	A reference natural-language answer; enables `applicable_when: reference_answer_text` judge criteria
`reference_source`	No	—	Provenance of the reference: `human_verified`, `independent_query`, or `unverified`
`expected_views`	No	—	Views the agent should touch; enables the `expected_view_recall` metric

reference_values entries:

Parameter	Required	Default	Description
`metric`	Yes	—	Column name in the agent's final query results (or `row_count`)
`expected`	Yes	—	Expected value
`tolerance_pct`	No	`0.0`	Allowed relative deviation, e.g. `0.01` for 1%
`widget_index`	No	—	Which result row to read the metric from

Keep the held-out set as its own versioned, git-committed artifact (golden_set: evals/held_out.yaml) so each revision stays immutable.

Data-quality scope

dq_scope restricts which configured checks: count as data-quality attribution evidence for the suite. By default it is derived automatically: any check whose dataset overlaps a table the agent's SQL touched.

Configuration Includes

The new sections merge identically to existing ones when using includes:, so you can keep evals in their own file:

# main.yaml
version: 1
includes:
  - evals/agent_variants.yaml
  - evals/suites.yaml

Validate Your Config

weiser eval-compile evals.yaml                      # all suites
weiser eval-compile evals.yaml --suite my_suite     # a single suite

This validates the new sections, resolves all agent_variant/semantic_layer/datasource references, checks that the suite has goldens, and runs the multi-variable lint — without executing anything.

semantic_layers​

Generic SQL​

Cube.js​

agent_variants​

The entrypoint factory​

Ablation example​

eval_suites​

Arms​

Goldens​

Data-quality scope​

Configuration Includes​

Validate Your Config​