An agent that touches production should be as accountable as the people who do.
This is the record of building one — in 10 commits across nine sprints, by one person directing an AI through twelve named roles. It is a demonstration, not a company. Everything below can be checked against the repository, including the parts that went wrong.
Act one
Why this exists
AI agents are increasingly asked to do real operational work — read a system’s documentation, diagnose an incident, propose a fix. The obstacle is rarely capability. It is accountability: teams cannot get sign-off to let an agent near production because nobody can explain, afterwards, why it said what it said.
SANDSCOPE is a control plane for that problem. Its central behaviour is not answering questions well. It is refusing to answer when the evidence it found does not support an answer — and proving, per claim, which document each statement came from.
Anything can answer. The engineering is in knowing when not to.
The corpus, the incidents and the metrics are synthetic. The engineering is not — the services, the failover, the retrieval, the evaluation, the tests and the deployment all genuinely run. That distinction is stated on the site rather than buried, because a reader who discovers a simulation themselves discounts everything around it.
4.7%
False answers
budget 5.0%
0.808
Classifier AUC
baseline 0.609
423
Tests
24 files
30
Defects logged
7 severity 1
14
Decision records
$0
Infrastructure cost
enforced by a test
Act two
How it was built
Not by one person typing. By one person acting as Product Owner and Executive Sponsor, directing an AI agent that worked in twelve named delivery roles — Business Analyst, Architect, QA Lead, Security Engineer and the rest — under a written charter with one rule doing most of the work:
A role may not sign off its own work.
That constraint is why this produced a defect log instead of a demo. The QA role caught the engineering role reporting a 0% error rate that was actually 56.6%. The security role found a test that passed while the service it tested was switched off. The programme role found two entire sprints that had been worked and shipped without ever being formally opened.
All 30 defects are published, including the embarrassing ones, because a delivery record containing only successes is not evidence of anything.
Whose story do you want?
A project is never one narrative. Pick a role to follow their thread — what they owned, what they produced, and the one moment where their particular worry turned out to be the right one.
QA Lead
QA
Refuses to accept 'it works' without evidence — and is suspicious of evidence that is too convenient.
Owns
Test strategy, Definition of Done, defect triage
Produced
Test plan · Defect log · 18 logged defects
Logged all 18 defects in this project, and the sharpest pattern in the log is that a third of them were checks that reported success while testing nothing at all.
Their thread — 4 moments across the build
Sprint 2
A test suite that reported 0% errors when the real rate was 56.6%
The refusal gate was measured on 22 questions written by the person who built it. On a larger, harder set it got 150 of 265 wrong.
The most dangerous test result is a good one from a sample you chose yourself.
cited: D-001
Sprint 5
A defect that neither component's own tests could have found
A semantic cache served a stale answer to a correction retry. Both the cache and the retry logic had correct, passing tests in isolation — nobody had tested the two together.
This is the strongest single argument in the whole project for running the assembled system rather than trusting unit coverage.
cited: D-006
Sprint 7
Two guards that were built to confirm, not to detect
A rate-limit pen test sent 8 requests against a limit of 20 and treated any error as a pass — so a dead service reported as correctly protected. A README checker searched for values as substrings, so a changed number could hide inside an unrelated one.
The standing rule that followed: a guard is not trusted until it has been run against the defect it claims to catch and observed to fail.
cited: D-013, D-015
Sprint 8
A green pipeline that had never run the code under change
CI had no job that built the web application. A framework upgrade reported ten green checks while the production build failed on every page.
A check that exercises none of the code under review isn't weak evidence — it's negative evidence, because it makes a reviewer more confident than they should be.
cited: D-018
Act three
What it proves, and what it doesn’t
62 of 62 requirements are complete. The other 0 are marked planned, and the traceability matrix says so rather than rounding up — a build check fails if any requirement claims to be done while the test it names cannot be found.
What is genuinely demonstrated: a deterministic multi-provider router with bounded spend; hybrid retrieval with per-claim citation; an evidence gate whose thresholds are derived from stated error budgets rather than chosen by feel; a trained classifier and cross-encoder shipped as ONNX; a security pipeline that has caught real problems including in itself; and a governance process that produced eighteen logged defects instead of a clean-looking story.
What is not: real users, real traffic, and the operational history that only comes from running something for months. This is a demonstration built to production standards, and it says so.
What happens next
Finish the release gate. Penetration tests and the threat-model review against the deployed system rather than a local one — Sprint 8’s remaining stories, unblocked now that both halves are live.
Decide the scope honestly. 0 requirements are still open, and many were written for a product with real customers. The right move is an explicit decision to de-scope what does not serve the demonstration, so the matrix tells the truth instead of implying permanent incompleteness.
Observability and load. Every fault in the threat model should reach a documented response, and the free-tier ceiling should be measured so the failure mode is a refusal rather than a bill.