Evaluation environment · July 2026
Claims Audit Agent: one task, three orchestration frameworks
Extended thinking cut LangGraph recall from 1.000 to 0.600 while raising Vercel AI SDK precision from 0.714 to 0.938. Same model, same tools, same frozen data, opposite behavior by orchestration.
Why the finding is trustworthy
- I did not trust labeled data I had not generated, so the generator assembles only valid claims and then injects exactly-known defects across five classes: 10 unbundling, 9 unit violations, 8 upcoding, 7 duplicates, and 6 missing modifiers, over 200 claims.
- An independent engine re-derives every label from the claim and rules alone before any record reaches disk. Generation fails hard unless the engine agrees with the injected labels on every claim, so the dataset is correct by construction rather than by inspection.
- A 25-claim hand-labeled holdout is kept structurally out of the system's reach.
- A deterministic YAML rule engine runs in CI on every push at precision, recall, and F1 of 1.000, and serves as the gold standard the agent arms are graded against.
- All three arms run the same four typed tools on the same frozen data with the same model (qwen/qwen3-32b) through one gateway. The TypeScript arm reuses the identical frozen data files without re-encoding and emits JSON the Python harness consumes unchanged.
Results
- Deterministic engine, in CI on every push
- precision, recall, and F1 of 1.000
- Fabrication rate, every arm
- 0.000
- Citation validity, every arm
- 1.000
- LangGraph recall, thinking off then on
- 1.000 then 0.600
- Vercel AI SDK precision, thinking off then on
- 0.714 then 0.938
- claude-agent-sdk on the holdout
- 1.000 precision, recall, and F1, at about 18 times the cost per claim
What came back against me
- The LangGraph arm's 0.000 fabrication rate is structural, not earned. An evidence-check node drops any citation it cannot verify, so the zero holds regardless of what the model proposes. The two autonomous arms earn their zero from the model plus a strict schema, which is a weaker guarantee, and the writeup says so.
- The claude-agent-sdk arm is demonstrated on the holdout, not run at full scale. It spawns a fresh CLI subprocess per claim, which is flaky against a third-party endpoint and roughly 15 times slower. The trade is cost, latency, and stability, not correctness.