Artifact — evals over fine-tuning
How the eval loop from the Evals over fine-tuning case study runs in production: five dimensions scored separately, observability on every answer, and a human-in-the-loop where the stakes demand it. Nothing ships below the bar; every new miss becomes a new eval.
Did we fetch the right source?
Pass = answer cites the exact policy section it relied on. No citation = fail, even if the wording sounds right.
Does the answer stay inside the source?
Pass = every claim traceable to the cited source. Any invented detail = fail.
When unsure, does it route cleanly?
Pass = plain-language stop reason, what happens next, time estimate, full context handed to the human.
Can we see what the agent did?
Every answer logs query, retrieved sources with versions, confidence, policy path taken, and outcome.
Where do humans stay in control?
Low-confidence and high-stakes answers queue for agent review before or after send; each reviewed miss becomes a golden-set eval.
How one miss became one eval