AI model evals

"Best results, not just cheapest" — made permanent. Each task type is scored at Haiku, Sonnet and Opus against the live PURE canon, so routing is proven by numbers: bump a task up if the cheap tier underperforms, drop it down if the expensive tier isn't earning its cost. Smart routing saves Pure Dollars (PD) by using the cheapest tier that still passes.

sample eval scores — run the harness for live

Avg routed score

—

Flag: bump up

—

Flag: save (drop)

—

Task types

—

Nightly over poppy_evals; scores 0–100 per tier. Routing recommendation is auto-derived.

Task type	Haiku	Sonnet	Opus	Routed	Recommendation

Reads mcp_ai_evals over poppy_evals; run via mcp_run_evals. Recommendation logic: if the routed tier scores below its pass bar bump up; if a cheaper tier scores within 3 pts of the routed tier drop to save; else hold. Pairs with the AI Cost toggles.

Admin only