Published study · May 2026
Augmented Operational Decisioning with Ontology-Grounded Local Agents
A federation of agents on a self-hosted 70B open-weight model produced operations-grade decisions on a contractor's real data with zero fabrications across 51 held-out decisions, and no frontier API anywhere in the decision path.
Why the finding is trustworthy
- Two-database holdout. The agents reason over one database. Ground truth lives in a second database they never query, and the holdout window is everything after the worldview database's last timestamp. The answer key is out of reach by construction, not by convention.
- Every claim was written down as a hypothesis with an explicit numeric threshold before the run, then graded pass or fail against that threshold.
- Decisions were scored on a multi-axis rubric by independent raters. Inter-rater reproducibility came out at Cohen's kappa 0.95, so the scores reflect the rubric rather than one grader's taste.
- Cost and compose-reliability targets were set in advance and treated as pass or fail gates alongside capability.
- The harness is wired into CI. A build fails on a 10-point capability drop or a 10% p95 latency regression.
Results
- Fabrication-guard trigger rate
- 0% (0 of 51 held-out decisions)
- Strict capability (exact operator match)
- 66.7% (10 of 15), against a pre-registered 55% threshold
- Operations-defensible, independent judgment pass
- 86.7% one-pass approve
- Substrate-consistent rationales
- 100%
- Hallucinated entities
- 0
- Inter-rater reproducibility
- Cohen's kappa 0.95
- Compose reliability
- 600 of 600 across two model families (321 on 8B, 279 on 70B)
- Decision-path API spend
- $0.00
- Total compute including debugging
- about $8.50 against a $26.66 budget
What came back against me
- A divergence-from-control metric came back at 0% and was marked instrument-invalidated. The fixture's feedback hints had been authored to align with the control's default picks, so both arms reached the same answer for the right reason on each side. The metric was measuring the fixture, not the model. It was traced, documented, and given a surgical fix for the next run rather than quietly dropped.
- An autonomous tool-call-rate hypothesis failed as anticipated, and the claim was reframed rather than defended. The system is prompt-grounded executable retrieval with a human approving each proposal, not an autonomous command loop.
- This was a proof of concept on a single fixture, not a production benchmark. The capability number covers one task on one cohort and the sample is small at 15 eligible decisions.