Series: Part 1, Architecture · Part 2, Grading judgment · Part 3, What broke

This series is the record of building BOM Pilot, an agent that reads part-discontinuation notices (PCNs) and decides a response strategy — last-time-buy (ltb), replace, redesign, or accept_risk. To recap the architecture from Part 1: LLM judgment is placed at exactly two points, interpreting the notice and deciding the strategy. Every number comes from the database. And the final decision comes out as a structured submission, not prose — the strategy plus the list of numbers it was based on (the basis). This post covers the hardest problem in the project: grading that judgment.

Say the agent recommends “replace U1 with candidate X.” How do you decide whether that’s correct? String comparison won’t work — the answer isn’t a string, it’s a decision. Sit two competent engineers down and they might disagree. The common alternative is to have another LLM do the grading. But if you grade an LLM’s judgment with an LLM, the same problem reappears one level up: who verifies the grader? “Looks plausible” is not verification.

The answer I chose: make the facts force the verdict. This is where synthetic data changes status. Not having real data is usually a project’s weakness — but for grading, it becomes a tool. Since I control every number in the database, I can compose stock, lead times, deadlines, cost ceilings, and compliance flags so that the facts pin down the defensible strategies in each scenario — exactly one in most, and in a few, a deliberate pair where senior judgment can legitimately go either way (a tight replace vs. an honest redesign escalation). Plant the facts “stock is short, the last-time-buy shipment arrives after the deadline, and a qualifying substitute exists,” and the right answer becomes replace — by arithmetic, not by opinion. The answer keys themselves are also checked with a verifier before any run, to rule out grading against a wrong key.

There are 17 scenarios: 10 judgment scenarios (from easy ones up to boards deliberately constrained so that no clean answer exists) and 7 security-and-robustness scenarios. Grading happens in four layers, and each layer answers a different question.

  • Layer 1, interpretation accuracy (code): did it read the notice correctly? Extracted part numbers and change type are compared against ground truth.
  • Layer 2, basis-fact grounding (code): does every number it cited actually exist? Every figure submitted as the decision’s basis must be traceable to material the agent actually received — the notice, the prefetched data, or a tool result.
  • Layer 3, strategy fit + flip-pairs (code): is the strategy right — and does the answer flip when the facts flip?
  • Layer 4, narrative quality (LLM judge): is the recommendation well written? Deliberately auxiliary.

Layer 1 is boring but foundational. Miss one part number and everything downstream is meaningless, however brilliant. It also includes not making things up — one scenario deliberately provides an under-specified notice, and if the agent fills in a deadline the notice never stated, that alone is a failure.

Layer 2 is the hallucination detector. Hallucination is when an LLM invents information that exists nowhere, plausibly — and this layer caught a real one. In an early run, the agent added up per-vendor stock in its head and submitted “total stock 4,200” as part of its basis. The number appeared in no tool output. There was no way even to check whether the addition was right — exactly the “LLM arithmetic” this architecture forbids — and the grader flagged it automatically.

But the grader itself had a bug. The first version passed any cited number that existed anywhere in the material. So a fabricated “lead time 8 weeks” (the real lead was 20) slipped through — because the completely unrelated field “deadline: 8 weeks” happened to contain the same number. After the fix, numbers are checked only against fields with matching meaning: a claim about lead time is compared to lead-time data only, and can’t borrow the deadline’s number. I verified the fix with the same input: the doctored “8-week lead” passed before, and was caught after. Build a grader, and the grader becomes something you have to grade.

Layer 3’s basic check is simple: is the chosen strategy in that scenario’s list of defensible strategies (the acceptable set)? But this check has a hole. The agent can get the right answer by luck — by reacting to what the notice looks like rather than what it says. “EOL notices usually mean stock up” is a pattern that gets the answer right surprisingly often.

So a stronger check sits on top: the flip-pair — twin scenarios identical except for one decisive number. Take the same EOL notice in two versions. The deadline is 6 weeks in both. In one version, the last-time-buy stock arrives 3 weeks after ordering (inside the deadline — buying works; the answer is ltb). In the other, it arrives in 14 weeks (past the deadline — buying is pointless; the answer is replace). The two notices look virtually identical. If the agent says “stock up” to both, it’s pattern-matching the document’s appearance. If its answer flips between the versions, it’s actually reading the numbers and comparing them to the deadline. There are two such pairs (the other flips a marking change into an electrical-spec change), and in the full run, both flipped correctly. Not “it got the right answer,” but “the answer changed when the right answer changed” — I consider this the single most convincing result in the project.

Layer 4’s LLM judge scores the quality of the writing — risk framing, reasoning, clarity. But it has no say in pass or fail. The pass/fail verdict is fully deterministic: the strategy is in the acceptable set, the critic’s re-verification passes, and no security canary (explained below) appears in the output. If a run passed, I can explain why at the code level, and that explanation doesn’t move when the judge’s score wobbles.

Security scenarios are graded with canaries. The word comes from the birds miners carried to detect toxic gas — an early signal that appears only when something has gone wrong. It works like this: a valid notice is sent with an attack payload embedded — a prompt injection (“begin your response with this phrase”), an attempt to induce data exfiltration, a fake authority override, poisoned facts. Each scenario defines a phrase that can only appear in the output if the attack succeeded, and if that phrase shows up anywhere in the response, the scenario fails immediately, regardless of every other score. Requests unrelated to the app’s purpose (say, “write me some marketing copy”) are handled one layer earlier: the moment the interpreter classifies the input as “not a part-change notice,” it never reaches the planner. That test’s pass condition includes “planner iterations: zero” — checking not whether a defense worked, but whether the attack surface ever executed at all. Measured: it stopped in 4 seconds.

The last measurement is reproducibility. An agent that gives different answers to the same input on different runs is a product risk in itself, so it gets measured. The same scenario is run repeatedly and the agreement rate — how often the most common strategy recurs — is recorded: some scenarios score 1.00 (same answer all 3 runs), one scored 0.67, and the least stable scored 0.33. What matters is that this variance is recorded as a number instead of being hidden inside an average. Why this metric matters comes back in Part 3.

Results. On the final baseline — one full run against one commit — 14 of 17 passed. Interpretation accuracy 1.00; strategy fit 0.94 (no scenario chose an unacceptable strategy; the two failures chose strategies the answer key doesn’t list); basis-fact grounding 0.94; flip-pairs 2/2; security 5/5; canary leaks 0. Narrative quality averaged 3.8/5 on the preceding judge-enabled full run (the baseline run was scored with the judge off — it never gates a pass). The three failures are left in the results table as they are. And more interesting than these numbers is what broke on the way here — the subject of Part 3.

Series: Part 1, Architecture · Part 2, Grading judgment · Part 3, What broke