How it is graded
How an AI model is graded on a made-up company's past invoices.
74 s Narrated by a synthetic voice. Every word is also on screen.Read it instead
On screen: How it is graded. A made-up company. Real runs. Fictional company
Voice: This is how PlatAtlas grades an AI model on your team's past answers.
On screen: Larkspur Garden Co. · accounts payable · 6 waiting. Harlan Irrigation Supply $7,984.56; Harlan Irrigation Supply $10,792.35; Northgate Grounds & Glasshouse $2,484.52; Amberline IT Services $4,071.33; Tessary Office Supply $1,914.08; Harlan Irrigation Supply $1,701.50. Fictional company
Voice: A made-up garden company gets the same kinds of invoices every week, and every past one has a recorded answer.
On screen: The example record as files, the earlier record the stand-in was checked on: CONTEXT.md, RUNBOOK.md, obligations.md, questions.yaml, thresholds.yaml, people.yaml, harness.toml, decisions/README.md, eval/README.md, eval/examples.csv, queue/README.md. The past answers file, eval/examples.csv, opens at its header and first two rows, literally in the file. This record's past answers were written by its rule, not by people. Fictional Illustrative
Voice: How they answer is written down in plain files the company keeps. Its past answers become the test.
On screen: The sample inbox's own record, a later one for the same company. 60 cells, sorted: right answers first. 2 of 60 missed · 3.3% · limit 5% · PASS. Measured 2026-09-23 · Claude Sonnet 5
Voice: Claude Sonnet 5 answered sixty past invoices. It missed two. The limit was five percent, so it passed.
On screen: Beside it: an earlier record for the same company · a deterministic stand-in, not an AI model. 34 of 60 missed · 56.7% · limit 5% · FAIL. Measured 2026-09-22 It answered pay every time. Its misses: possible-duplicate 8 · price-variance 11 · vendor-change 5 · hold-for-receipt 10.
Voice: On an earlier record for the same company, a stand-in with no AI model answered pay for every invoice. It missed thirty-four, and the check failed it.
On screen: The steward's words for 'paid': "None of the checks applies: the remit-to account matches the vendor master and that account has not changed in the last 30 days; ..." Fictional Example wording Claude Sonnet 5: 2 missed, then 0 missed. Claude Haiku 4.5: 13 missed, then 8 missed, FAIL. Measured 2026-09-23 Our own runs. platatlas.com runs TypeSafe.
Voice: Write down what each answer means in the steward's own words, and the same models miss less. The weakest one still fails.
On screen: Paid · signed by Dana, AP lead. Declared Illustrative A new row joins eval/examples.csv: new-01,cases/new-01.json,pay,dana,2026-09-23,declared,"signed in the queue"
Voice: Every answer is written down with who made the call, and joins the test.
On screen: The AI is rented. Your team's know-how is owned. platatlas.com · platatlas.com/demo/try. Larkspur Garden Co. and Dana are fictional; the invoices are made up. The checks are real runs. Switching models runs on your own server today. Narrated by a synthetic voice.
Voice: The AI is rented. Your team's know-how is owned.
What's real in this film
- The six invoices are the sample inbox's own cases, on a made-up company.
- The passing check is the sample inbox's own, run 2026-09-23 with Claude Sonnet 5: 2 of 60 missed.
- The failing check is a deterministic stand-in with no AI model, on an earlier record for the same company, run 2026-09-22: 34 of 60 missed.
- Before and after: our comparison of the steward's own words, run 2026-09-23. Claude Sonnet 5 went from 2 to 0 missed; Claude Haiku 4.5 went from 13 to 8 missed.
- The file names and the past answers rows are the example record, shown as they are in its files.
- Illustrative: the signature stamp and the new row. Dana is fictional.