Model Evaluation & Optimization

Multi-provider model system

Nova's agent platform connects to thirteen LLM providers simultaneously. Providers and their models are registered in apps/admin/src/lib/agent-core/models/catalog/ — one TypeScript file per provider. Adding a new model is one entry in the provider file; adding a new provider is a new profile file plus a line in catalog/index.ts.

Provider catalog

Each model is described by a CatalogModelEntry with id, tier (fast | balanced | powerful | custom), pricing [input, output], contextWindow, maxOutputTokens, supportsReasoning, and supportsImageInput.

Model ID namespacing

Provider-specific models use a provider/model namespace to prevent catalog collisions. The routing table in selector.ts → getProviderForModel() uses prefixes to determine the provider:

• vertex/… → Vertex AI
• cohere/… → Cohere
• cerebras/… → Cerebras
• fireworks/… → Fireworks
• together/… → Together
• mistralai/… → Mistral (dedicated provider, not OpenRouter)
• <org>/<model> (any remaining slash) → OpenRouter

Anthropic models (claude-*) have no prefix. OpenAI models (gpt-*, o\d*, *codex*) have no prefix.

OpenRouter as an escape hatch: any model string containing a / that does not match a dedicated provider prefix is routed through OpenRouter. This lets you reference any of OpenRouter's 300+ models without adding a catalog entry — just supply the full org/model slug, for example qwen/qwen2.5-72b-instruct.

Capability gates

Before routing to a model, the system checks provider capabilities (ProviderFeatureId):

• image-input — required when images are attached
• extended-thinking — required when extended reasoning is requested
• structured-output-guaranteed — required for strict JSON schema output

If the preferred model's provider lacks a required feature, the router falls back to the next tier up.

The Auto router

When an agent's model is set to Auto, the routeRequest function in router/auto-router.ts picks a model at runtime via a two-step process.

Step 1 — Complexity classification

A cheap, fast classifier LLM (default: Haiku) is called with the last user message (truncated to 500 chars) to classify the request into one of three tiers:

The classifier is a single LLM call with maxTokens: 5 — cost is negligible ($0.0001 or less).

Step 2 — Route once per thread

The result is locked on the thread after the first turn. Subsequent turns reuse the same model, which preserves prompt-cache hits and server-side compaction across turns. Never swap models mid-thread. Callers must persist the routed model to the thread record and pass it as modelOverride on subsequent turns.

Fallback chain

1. Classified tier, if the preferred model is connected and satisfies required capabilities
2. Next tier up (capability upgrade), if required features demand it
3. routingConfig.defaultModel, if no eligible model exists

Skip conditions: the classifier is skipped and defaultModel is used immediately when:

• Only one provider is connected (no routing decision needed)
• The classifier call times out (default 3 s) or returns an unrecognized token
• connectedProviders.size ≤ 1 (Vertex uses ADC and is not counted by key-presence check)

Configuring Auto

AutoRoutingConfig lets you override the tier-to-model mapping:

{
  defaultModel: "claude-sonnet-4-6",
  autoTierModels: {
    cheap: "claude-haiku-4-5-20251001",
    mid: "claude-sonnet-4-6",
    frontier: "claude-opus-4-6",
  },
  autoClassifierModel: "claude-haiku-4-5-20251001",
  autoClassifierTimeoutMs: 3000,
}

Eval harness — CLI scripts

The eval harness lets you replay past runs against candidate models to measure quality, cost, and latency before committing to a model swap. Both scripts live in apps/admin/scripts/.

Two corpora

Running a replay

Both scripts must be run from apps/admin/ with the react-server export condition to bypass server-only imports:

cd apps/admin

# Flow-run corpus: compare Haiku and Sonnet on the support-responder agent
bun --conditions react-server scripts/_eval-model-replay.ts \
  --agent support-responder \
  --models claude-haiku-4-5-20251001,claude-sonnet-4-6 \
  --n-runs 5 \
  --budget-cap 5.00 \
  --persist

# Thread corpus: compare on recent support-responder conversations
bun --conditions react-server scripts/_eval-thread-replay.ts \
  --agent support-responder \
  --models claude-haiku-4-5-20251001,claude-sonnet-4-6 \
  --limit 10 \
  --budget 5.00 \
  --persist

Flags

Budget hard-stop

The pre-flight estimate (estimateBatchBudget) is advisory only. The authoritative stop is a running cumulative actual spend tracker: after each completed (run × model) pair, the real cost (agent run + judge) is accumulated. The next iteration checks isBudgetCapReached(cumulativeActualSpend, budgetCapUsd) before starting. This means the cap fires on real spend, not estimates, which matters because the upfront estimator uses conservative token baselines (6k input + 1k output) and can under-count heavy agents.

Lesson learned: an earlier version of the estimator significantly under-counted actual spend on the support-responder agent (which has large context from tool history). Use --limit to keep batches small, and rely on the hard-stop rather than the estimate.

Write-denied safe runner

All eval replays enforce a strict deny-list of external write actions via toolPolicy.deny. No replay can create a Linear issue, send an email, post to Slack, create a GitHub artifact, write a database record, or delegate to another agent. The canonical deny-list is in safe-replay-runner.ts → DENIED_WRITE_ACTIONS.

Gmail draft writes get an additional mock: installGmailMock() intercepts gmail.googleapis.com POST requests to /drafts and returns a stub response, so the Gmail draft action "succeeds" in the replay without creating a real draft. All other traffic passes through.

Judge rubric for write-denied replays: the thread-replay script appends a rubric note to the judge prompt explaining that side-effect writes are disabled. The judge scores quality of the response (correctness, relevance, reasoning, plan) — not whether the task was fully executed. A model that correctly describes what it would do and why scores well even without completing writes.

PII redaction

Before any eval output is passed to the judge or persisted to Firestore, redactPii() strips email addresses with a regex pattern. This is best-effort — do not store raw customer data in eval results.

Output — the comparison table

model                                   runs  avgScore  passRate  avgCost   p50ms   p95ms
claude-haiku-4-5-20251001               5     71.0      40.0%     $0.0032   2300ms  4100ms
claude-sonnet-4-6                       5     88.0      100.0%    $0.0890   8200ms  12000ms

Columns: model, runs (replayed), avgScore (0–100), passRate (% with score > 75), avgCost (USD per run), p50ms/p95ms (wall-clock latency).

What makes a run replayable?

isReplayableRun() filters to runs that are:

• Status succeeded or completed
• Not older than 30 days (stale trigger context is unreliable)
• No pausedReason set (HITL-paused runs require live human input)
• Not slack_events_api triggered without stored conversation history (would replay with no context)

system_model_optimizer — the weekly automated optimizer

The optimizer is a system flow (flowType: "system_model_optimizer") that runs on a cron schedule (0 2 * * * — 2 AM daily) and produces HITL proposals for human review. It never auto-applies a model change.

What it does

1. Scans all eval-enabled agents and flow agent-nodes that have enough recent replayable runs (default ≥ 10)
2. Replays those runs against each candidate model in the configured list
3. Scores replayed outputs with the LLM judge
4. Proposes a model swap only when: the candidate is cheaper AND the quality drop stays within qualityFloorDelta (default −5 points)
5. Saves proposals as FlowVersion records (for flow nodes) or agent proposal records (for agents)

Editable config

The optimizer reads its runtime knobs from config/model-optimizer via readEditableConfig. The code default is:

{
  candidateModels: ["cerebras/gpt-oss-120b", "mistralai/ministral-8b-latest"],
  qualityFloorDelta: -5,         // max allowed score drop vs baseline
  perRunBudgetUsd: 10.0,         // hard budget cap per optimizer pass
  threadsPerAgent: 10,           // runs to replay per target
  targetAgents: [],              // empty = all eligible agents
}

You can tune any of these by writing to config/model-optimizer in Firestore — no PR needed. The change takes effect on the next cron tick (the config is cached ~60 s).

Proposal format

A proposal contains:

• kind: "flow_node" or "agent"
• targetId: the flow or agent ID
• nodeId: (flow_node only) the specific node being overridden
• currentModel / proposedModel
• expectedScoreDelta: proposed − baseline average score
• deltaCostPct: % cost change (negative = cheaper)
• deltaLatencyPct: % p50 latency change
• basedOnRunCount / basedOnRunIds
• judgeReasoning: summary from the eval judge

For flow nodes, the proposal is saved as a modelOverride in a FlowVersion (applied per-node, not globally). For agents, it sets modelPreference. Neither is auto-applied. A human reviews and accepts or rejects the proposal via the version history UI.

Enabling the optimizer

The system flow seeds with enabled: false. To enable:

1. Find the system_model_optimizer document in flows/.
2. Set enabled: true.

Cron gotcha: the live cronAgentScheduler Cloud Function runs every minute and picks up any enabled cron flow immediately. Enable only when you are ready for the first pass to run. Treat the first pass as a pilot — review the proposals before accepting any.

Selection algorithm

chooseBestCandidate picks the candidate that:

1. Is cheaper than the current model (strict requirement)
2. Scores ≥ baseline.avgScore + qualityFloorDelta (default: baseline − 5)
3. Among those, picks the cheapest; breaks ties by p50 latency, then by highest score

Running an ad-hoc eval and acting on results

Workflow

1. Dry run first — use --dry-run to see which runs will be replayed and the estimated cost before spending anything.

2. Start small — use --n-runs 3 or --limit 5 on the first pass to sanity-check costs and judge scores. The budget estimator under-counts for heavy agents.

3. Read the comparison table — look at passRate (% > 75) as the primary quality signal. avgScore alone can be misleading if a model scores 74 on most runs (just below the pass threshold) while another scores 90 on a few.

4. Act on proposals — proposals go into version history. To apply a flow-node proposal: open the flow builder, navigate to the node's version history, and accept the proposal. To apply an agent proposal: open the agent editor and update modelPreference.

5. Never auto-apply — the system deliberately surfaces proposals to humans. An eval replay scores quality on a sample of past runs; real production traffic has variance the sample may not capture.

Interpreting scores

Pass threshold is 75 (PASS_SCORE_THRESHOLD in model-eval.ts).

Pilot findings

From the initial eval pilot on the support-responder agent across 5 flow runs:

• Sonnet 4.6 was the clear quality and reliability leader: avgScore ~88, pass rate 100%
• Haiku 4.5 was ~30× cheaper and ~3.5× faster but scored ~71 with 40% pass rate — adequate for simpler queries, unreliable for complex support cases
• Cerebras GPT-OSS 120B showed comparable throughput at $0.35/$0.75 per 1M tokens — promising for latency-sensitive flows
• Mistral (free-tier API) hit 50k token/minute rate limits (429s) under batch replay — not suitable for the optimizer's concurrent replay loop

Recommended tiering strategy:

• Use Sonnet (or Auto routing) as the default for customer-facing agents where quality matters
• Use Haiku or Cerebras GPT-OSS 120B for internal utility flows (classification, extraction, summarization) where a 20-point score gap is acceptable
• Use the eval harness to validate before switching any production agent — do not rely on benchmark scores alone