SANDSCOPE / HANDOVER
Everything, at whatever depth you need
One document for two readers. The prose reads end to end in plain language and assumes nothing — that is the whole handover for someone who does not write code. Wherever there is more to say, a deeper block underneath carries the parameters, the formulas, the tradeoffs and the failure modes.
Writing for the middle would patronise one reader and starve the other. Writing two documents would guarantee one of them goes stale. Nothing below is hidden from search or from a screen reader — the disclosure is native, and works with JavaScript off.
One
What this is, and who it is for
SANDSCOPE answers questions about a company’s production software — the kind an on-call engineer asks at 3am. Why did the database slow down? Is this change safe to ship? It reads a fixed library of internal documents, finds the relevant passages, and answers with a reference beside every claim, like footnotes in an essay.
When the documents do not actually answer the question, it says so. That refusal is the product.
The company is invented — every service, document and incident was authored for this project, and the data page publishes all of it. The engineering is not invented: the services run, the tests pass, and both halves are deployed.
deeperWhy an agent needs this at all, and what it costs to skip
The obstacle to putting an agent near production is rarely capability — models are good enough to read a runbook and reason about a metric. It is accountability: nobody can explain afterwards why it said what it said, so nobody can sign off on it going live.
Three things have to be true before that sign-off is possible, and this system exists to make all three checkable. Every claim traces to a passage. The system declines rather than filling a gap. Every run leaves an inspectable record of what it did, what it cost, and where it stopped.
The alternative failure is specific and expensive: a confident, well-cited, entirely wrong answer about a service the evidence was never about. That is the case the evaluation set is deliberately loaded with — see the 214 questions described below.
Two
The refusal decision
Before answering, the system scores whether what it found is good enough. The score falls into one of three bands, and it behaves visibly differently in each: it answers with citations, it answers but flags the answer as partial, or it refuses and produces no draft at all.
Measured over 715 questions where the right answer is known: it wrongly answers 15/319 questions it should refuse (4.7%, against a 5.0% budget) and wrongly refuses 9/396 it could have answered (2.3%, against 10.0%).
Those budgets are deliberately asymmetric. A wrong answer about production is far more expensive than an unnecessary “I don’t know”, so the system is tuned to make the second mistake rather than the first.
It answers, but flags the answer as partial rather than confident.
Routes to an explicit adjudication step. Never silently upgraded to sufficient — reading ambiguous as a soft yes is how a three-state assessment collapses back into the two-state one it replaced.
answerableWhat is the connection pool ceiling for orders-db?
the hard partThe highest-scoring question here is one the corpus cannot answer. It beats two questions that genuinely can be. No single threshold separates them, which is why the middle band exists and why 87% of questions land in it.
and the cost of itNot one of these six is refused outright. Even the DNS question — a topic the corpus never mentions — scores 1.57 against a 0.74 floor, so it defers. Drag below 0.74 to see a refusal; nothing real lands there.
thresholds: retrieval/evidence.py · scores measured against the live corpus
deeperThe actual signals, thresholds, and why they are multiplied
No single retrieval signal separates the two classes. Measured on the first 20 questions: dense similarity gave answerable-min 0.268 against unanswerable-max 0.270 — a margin of minus 0.003. Term coverage was worse at −0.333, because “what is the data retention obligation” scores 1.00 against a sentence saying retention is explicitly not covered. Every query term present, inside a disclaimer.
The fused, normalised score is useless here by construction: normalising within a result set makes the top hit 1.0 whether the match was excellent or hopeless. A prior system read 0.031 on both classes — structurally incapable of refusing, at any threshold.
So the gate uses the product of unnormalised dense cosine and unnormalised BM25, not their sum. Multiplication requires both signals to be present: a high lexical score from one repeated identifier, with no semantic proximity, cannot clear the bar alone.
| Constant | Value | What it does |
|---|---|---|
| INSUFFICIENT_BELOW | 0.74 | Below this, refuse outright |
| SUFFICIENT_ABOVE | 10.38 | Above this, answer |
| MIN_TERM_COVERAGE | 0.30 | Too few query terms present — refuse before scoring |
Between 0.74 and 10.38 the verdict is ambiguous and routes to an explicit adjudication step. That band is wide on purpose: the overlap is real and is reported rather than tuned away. It is also the system’s biggest live weakness — 87% of questions land in it.
One further guard sits on top. A question demanding a specific value (“how long”, “what is the limit”) whose retrieved passages contain no value is downgraded to ambiguous rather than answered — never straight to refusal, because it is a heuristic about question shape and a heuristic is not entitled to refuse on its own authority. It exists because “how long is the observation period between regions” scored 8.85 against a passage that says an observation period exists and never says how long.
apps/agent/sandscope_agent/retrieval/evidence.py
Three
What it knows, and what it deliberately does not
The library is 19 documents — 5,759 words of runbooks, policies, postmortems and architecture notes — split into 87 passages. Small on purpose: a corpus you can read in an afternoon is one where you can check the answers yourself.
The invented company has 19 services in 4 criticality tiers with 27 dependencies between them, and 8 things that can go wrong, each with the runbook that covers it.
The thresholds above come from 715 questions where the correct behaviour is known in advance — 396 answerable, 319 deliberately not. Crucially, nobody labelled them by hand. An answerable question is generated from the passage that answers it, so the passage is the answer by construction.
deeperHow each question is manufactured, and the trick that matters
| Count | Mechanism | Label |
|---|---|---|
| 214 | property documented for another entity | unanswerable |
| 152 | symptom from fault signal | answerable |
| 127 | question from any section heading | answerable |
| 93 | section from runbook heading | answerable |
| 86 | entity cannot exhibit this fault | unanswerable |
| 24 | policy value lookup | answerable |
| 19 | topic absent from corpus | unanswerable |
The 214 matter more than the rest combined. They take a property the corpus documents for one service and ask it of another — “what is the connection pool ceiling for events-bus”, when the ceiling of 100 is documented for orders-db and nothing else. Retrieval returns a real passage with a real number about the wrong subject. A system without an evidence gate answers confidently and is completely wrong, and nothing about the answer looks wrong.
Absence is verified, not assumed. Gap questions are dropped at generation time if the corpus turns out to contain the terms after all. That check exists because the gap list was wrong once: it claimed the corpus could not say who approves an emergency change. It can. The gate scored it answerable, the author had marked it unanswerable, and the author was wrong — recorded in the open rather than deleted, because a gap list is itself a claim about the corpus.
Without verified gaps, “correct refusal” cannot be measured at all: every refusal counts as a mistake, and the threshold could be set to zero with no test noticing.
apps/agent/corpus/GAPS.md · sandscope_agent/evaluation/dataset.py · the full inventory
Four
How it is put together
Four pieces, each with one job.
- The website — what a visitor sees, and the only thing holding the password to the AI service.
- The agent — the thinking: find evidence, judge it, draft, check its own citations.
- The memory — the documents, and a record of every past run.
- The models — five external AI providers in a fixed order, with automatic failover.
The browser never talks to the agent directly and never holds its credentials. Everything goes through the website’s own server, which is the only component that knows where the agent lives. The architecture page draws the request path, and every box on it links to the file that implements it.
deeperThe orchestration graph, node by node
One compiled graph, 12 nodes, and every workload runs on it. Incident triage, change review and postmortem drafting differ only in data — a query builder, a system prompt, and a deterministic risk scorer. A second graph would disprove exactly the workload-agnosticism the first is meant to demonstrate.
classify → retrieve → assess_evidence ⟨refuse ∣ adjudicate ∣ hypothesise⟩
hypothesise → verify ⟨hypothesise ∣ escalate ∣ propose_action⟩
propose_action → risk_gate ⟨await_approval ∣ emit⟩
Four terminal states: refuse, escalate, await_approval, emit. The important one is the third — await_approval has no outgoing edge to any node that does work. A gated run is never resumed; approving it creates a new run pointing back at the gated one, so the pause is permanently in the record rather than erased by the decision (ADR-0006). A test enumerates the compiled graph’s edges to assert this, because reading the builder only proves what the author intended.
The verify → hypothesise edge is the citation loop. If a claim carries no resolvable citation, the draft goes back with feedback. An earlier version re-sent an identical prompt, so the retry could never succeed — the edge existed, the feedback did not (D-005).
apps/agent/sandscope_agent/orchestrator/graph.py
deeperRetrieval: parameters, weights, and the degraded path
| Component | Setting |
|---|---|
| Lexical | BM25, k1 = 1.5, b = 0.75 |
| Dense | 768-dimension hashing embedder, cosine |
| Fusion weights | lexical 0.6 · dense 0.4 |
| Chunking | heading-aligned, min 200 chars, merged below that |
| Default depth | top 6 |
Lexical is weighted higher because identifiers carry more diagnostic information than phrasing in this corpus — BM25’s separation between the two classes measured better than 3× the dense signal’s.
When the embedder is unavailable, retrieval degrades to lexical-only and says so. A degraded run is marked ambiguous rather than scored: the combined score is a product, so a missing dense term makes it zero, and zero would read as “weak evidence” when it means “unresolved evidence”. Those are different states and conflating them is how a system quietly returns worse answers while reporting normal operation.
retrieval/bm25.py · retrieval/embedding.py · retrieval/hybrid.py
Five
What happens when things go wrong
Three habits repeat throughout, and each exists because of a specific failure that already happened here.
- It fails safe, not open. If the traffic limiter cannot be reached, requests are refused rather than allowed through. An outage becomes unavailability, never unlimited free AI for whoever finds the URL.
- It reserves money before spending it. Every model call is costed and reserved before it goes out. No budget, no call.
- Risky actions stop for a human. Anything touching production halts and waits.
You can test the failover yourself on the console: tick a provider off, run a question, and watch the trace route around it.
deeperRouter, cache and spend guard — with the defect behind each
Router. Fixed order: groq → gemini → cerebras → openrouter → mistral. A rate-limited provider is disabled for a bounded interval, and the clock is injected so expiry is tested rather than waited on. Failure injection is scoped to a single run — the router is built inside the stream handler and discarded when it ends, so no visitor can degrade another’s run (T-6).
Spend guard. Reserves against the most expensive provider that could still serve, before the call. Pricing the first candidate under-reserved by 4× the moment failover reached a costlier model (D-010) — the guard had assumed the cheapest option would serve. Default ceiling RUN_BUDGET_USD = 0.02; a zero ceiling now prevents startup rather than killing every run mid-stream with a stack trace (D-007).
Semantic cache. Exact hash first, then vector similarity. Its threshold belongs to the embedder rather than the cache — a module-level constant was wrong in both directions at once (ADR-0008). The defect worth knowing is D-006: the cache served the previous answer to a correction retry, at 0.886 similarity against a 0.60 threshold. Both components were individually correct and individually measured. The cache’s test asked whether two different questions collide (0.208 — correctly no). Nobody asked whether a prompt collides with its own correction until the two were wired together and run.
router/router.py · router/cache.py · orchestrator/budget.py
deeperApplied ML, and the model that is deliberately switched off
| Artefact | Result | In the live path? |
|---|---|---|
| Sufficiency classifier | AUC 0.808 vs 0.609 baseline | no |
| Cross-encoder re-ranker | chunk MRR 0.528 → 0.587, p50 18.5ms | trained, not wired |
The classifier is the most instructive thing in the project. It is trained, calibrated, ONNX-served, tested — and not used. Wiring it in was measured rather than assumed: as a hard gate it refuses 56.8% of answerable questions against a 10% budget, because its threshold was calibrated for a false-answer budget and is properly conservative as a probability, catastrophic as a binary decision.
A tuned two-sided band looked excellent — deferral fell from 87.3% to 1.3% with both error rates apparently unchanged. Then it was scored on folds it had not been tuned on: 6.1% false answers against a 5% budget, and 2 of 5 folds could find no workable band at all. The improvement was the band memorising the questions it was graded on.
That is D-001 in different clothes, and much harder to spot: 715 is a real sample, a sweep is a reasonable thing to do, and the result was plausible in size and direction. It would have shipped as genuine. The one-line lesson: an operating point chosen on the data it is then scored against is not a measurement, it is a memory of that data.
Both models are trained offline and served as ONNX with no training framework in the serving image (ADR-0009). That turned out to be a security boundary too — the training extra carries four known RCE advisories while the runtime closure audits clean.
ADR-0013 · training/evaluate_classifier_as_gate.py
Six
Security posture
The endpoint is public, and it spends money on someone else’s API when used. Those two facts together are the whole threat picture: the realistic attack is not data theft, it is running up a bill or degrading the demonstration for other visitors.
The strongest control is architectural rather than defensive: no tool executes against any real system. Remediation is text. The worst outcome of a successful prompt injection is a wrong answer, not a wrong action.
deeperThreat model, controls, and the residual risks that are accepted
| Threat | Control |
|---|---|
| T-1 unbounded spend | Per-IP sliding window, daily token ceiling, spend guard — all fail closed |
| T-3 runtime called directly | Bearer token on every route, constant-time comparison |
| T-4/T-5 prompt injection | Corpus immutable at runtime; retrieved and user content both delimited, neither granted instruction authority |
| T-9 IP correlation | Salted digest only; no raw address persisted anywhere |
| T-11 model triggers an action | No tool acts externally. Risk-gated actions additionally require terminal human approval |
| T-12 token reaches the browser | Read only in route handlers; a test greps the built client bundle and fails on a match |
| T-16 secret in a trace attribute | Span exporter uses an allowlist — anything else is dropped, not redacted, because a redaction that fails is invisible |
Accepted residual risks, stated rather than closed. Session identity is a cookie: it scopes memory and binds approvals, and it is explicitly not authentication (T-17). SSE connection exhaustion is Medium (T-14). Prompt injection through the incident body is Medium and accepted, on the grounds that no tool can act (T-15).
The pipeline runs CodeQL, Semgrep, Trivy, gitleaks, pip-audit, npm audit, a CycloneDX SBOM, OWASP ZAP and six scripted penetration tests. It has found real problems including in itself — a shell injection in a workflow input, an unbounded request body, and a rate-limit test that passed while the service was down because its condition accepted any status ≥ 400 and could not tell “refused correctly” from “not running” (D-013).
docs/05-security/THREAT_MODEL.md
deeperData model, and what is persisted
17 tables. The estate and corpus: service, service_dependency, telemetry_event, incident, document, chunk, chunk_embedding. The record of what happened: session, memory_item, run, span, citation, approval, cache_entry, provider_event, spend_ledger, eval_run.
Citations are stored per claim rather than as a blob, so a reviewer can ask which passage supported a specific sentence months later. A citation whose chunk does not exist cannot be stored at all — it is a foreign key, and that constraint is the point: a citation pointing at nothing is not evidence.
The runtime holds no persistent local state. The retrieval index is rebuilt at startup rather than cached to disk, which is milliseconds at this corpus size and means a container can be replaced without losing anything (NFR-005).
apps/agent/migrations/0001_initial.sql
Seven
How it was built, and how that is proven
By one person directing an AI through a written charter of 12 named delivery roles, across nine sprints. One rule does most of the work: a role may not sign off its own work. The council page publishes the roster and all 24 role reactions to real defects and decisions.
All 30 defects are published, including the embarrassing ones — 7 of them severity 1. A delivery record containing only successes is not evidence of anything.
The governance is enforced rather than described: a requirement claiming Done while the test it names does not exist fails the build. 62 of 62 are Done under that definition.
deeperThe guards, and what each one caught
Eight checks run on every push, and their unusual property is that they can fail the build over documentation, not just code.
| Guard | Catches |
|---|---|
| check-traceability | A requirement claiming more than the repo can show (D-014) |
| check-readme | Any figure in the README that disagrees with the derived record (D-015) |
| check-deploy-claims | A document asserting the app is undeployed after it went live (D-019) |
| check-workflow-shell | A comment inside a backslash continuation silently truncating a command (D-011) |
| check-sprints | A sprint number used before its plan exists (D-016) |
| check-docs / check-config / check-secrets | Stub artefacts, config drift, credentials in a diff |
Each was verified by breaking something and confirming it noticed. That is not ceremony: the first README checker could not fail. It searched for each figure as a substring of the whole file, so changing “Commits | 54” to 99 still passed, because “54” appears in “54% of questions” (D-015). A guard that has only been run against a passing tree has not been tested.
The most expensive gap was the reverse direction. The traceability guard catches over-claiming and nothing caught under-claiming, so 20 requirements sat at Planned while already implemented and passing (D-020) — found only by an explicit audit. That asymmetry is still open.
scripts/check-*.mjs · apps/agent/tests/test_guards_fail_on_bad_input.py
deeperThe defects worth knowing, and what class each belongs to
| ID | What | Class |
|---|---|---|
| D-001 | 0% false-answer rate reported at a gate; real rate 56.6% on 534 questions | Test set written by the implementer |
| D-006 | Cache served the previous answer to a correction retry | Emergent interaction between two individually correct components |
| D-013 | Rate-limit pen test passed while the service was down | A test written to pass rather than to detect |
| D-018 | CI never built the web app; a major framework bump reported 10/10 green while broken | A pipeline whose green tick exercises none of the code under change |
| D-021 | Landing page rendered blank without JavaScript, for four sprints | Motion implemented in JS when CSS already did it |
| D-025 | Local tests passed against a stale build, twice — masking D-024 entirely | A green run that exercised the previous binary |
The distribution is the finding: not one defect was caught by code review, and not one by a unit test written before it. Every one was caught by executing something — the assembled system, a measurement over a large labelled set, a container in CI, or a browser.
docs/04-quality/DEFECT_LOG.md · the delivery record
Eight
Running it
Everything runs offline. No API keys, no database and no accounts are needed to clone it and run the full test suite.
# the agent and its tests
cd apps/agent
python -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest tests/ -q
# the website: unit, browser, and a local copy
cd apps/web
npm ci
npm run test:unit
npm run test:e2e
npm run devIt costs nothing to run. Every dependency sits on a free tier, and a test fails if a paid host ever appears in a deployment manifest.
deeperDeployment topology, and the operational gotchas
| Component | Where | Note |
|---|---|---|
| Website + BFF | Vercel, dub1 (Dublin) | Deployed from apps/web; not connected to the repo, so publishing is manual |
| Agent runtime | Northflank, London | Container binds 7860, 0.0.0.0 |
| Postgres + pgvector | Neon, Ireland | Same region as the functions — cross-region egress is how a free tier stops being free |
| Redis + Vector | Upstash, Ireland | Rate limiting and the semantic cache |
ADR-0003 placed the runtime on Hugging Face Spaces “because it is free” and was never checked against the pricing page. Docker Spaces are PRO-only, so three sprints of deployment work targeted a platform that could not host it at $0 (D-017). The cost was not a subscription — it was rebuilding the deployment decision at the release gate. The charter now requires an ADR depending on a third party’s pricing to name the page it was read from and the date.
Two smaller traps worth carrying forward. .env values are quoted; strip the quotes before pasting into any hosting UI — that caused an auth mismatch across eight secrets at once. And Vercel enables Deployment Protection on new projects by default, which 302s every route to a login wall.
ADR-0012 · deploy/Dockerfile · apps/web/vercel.json
Nine
Running the session
Thirty minutes, with the repository open. Show before you explain — watching it decline is more convincing than any description of it declining.
- 0–3What it is. Section one, out loud. No code yet.
- 3–8Show it refusing. Console → the preset that the corpus cannot answer.
- 8–13Show it recovering. Break a provider, run again, watch the trace route around it.
- 13–18Show the data. The data page — especially what the corpus deliberately omits.
- 18–24The defect log. Land on D-001 and D-021. This is where credibility actually sits.
- 24–28Run the tests. One command each side. Watching them pass does the work of a lot of talking.
- 28–30What is still open. Ending on the honest gap is the most in-character thing available.
Questions you will be asked
“Is the data real?” No, and the site says so on its own page. The engineering, tests, deployment and measurements are real. Anyone who discovers a simulation themselves discounts everything around it.
“Why does it refuse so often?” It defers on 87% of questions, which is the system’s largest weakness and is published as one. The signals genuinely overlap; the alternative is answering confidently on evidence that does not support it.
“What would you do next?” Close the one-directional traceability guard, finish release hardening, and generate the last hand-typed document. Naming the gaps unprompted is the entire point.
Ten
Glossary
- Agent
- A program that uses an AI model to work through a task in steps rather than answering in one shot.
- Citation
- A pointer from one sentence to the exact passage supporting it. Pointing at nothing means the sentence is not evidence.
- Retrieval
- Finding relevant passages before answering. Two methods run together: keyword matching and meaning-based matching.
- BM25
- The standard keyword-ranking algorithm. Favours rare terms and short documents.
- Embedding
- A list of numbers representing a piece of text's meaning, so similarity can be measured arithmetically.
- Evidence gate
- The check deciding whether what was found is good enough to answer from. The heart of the system.
- Failover
- When one provider fails, automatically moving to the next in a fixed order.
- Fail closed
- When something breaks, refuse rather than allow. The opposite turns an outage into a bill.
- Semantic cache
- Remembering answers so a rephrased question does not cost another model call.
- ONNX
- A portable model format, so a model trained with heavy tooling can be served without it.
- AUC
- How well a classifier separates two classes. 0.5 is a coin flip, 1.0 is perfect.
- Cross-validation
- Tuning on part of the data and scoring on the part not seen. The difference between a measurement and a memory.
- ADR
- Architecture Decision Record — what was decided, why, and what it cost. Never edited; a reversal is a new one.
- SSE
- Server-Sent Events. A one-way stream letting the browser watch a run happen rather than wait for a result.
- BFF
- Backend For Frontend. The server layer holding credentials the browser must never see.
- Traceability matrix
- The table linking each requirement to the test proving it. A false claim fails the build.