SANDSCOPE / RELIABILITY
Measured, including where it fails
An agent that answers everything is not useful; the hard part is knowing when to decline. These are the measured error rates of that decision, the thresholds that produce them, and the checks that are still failing. The failing ones are published because a reliability page listing only successes is the same mistake as the gate that once reported a 0% false-answer rate on a sample of 22.
Built for a laptop display — everything here works on a phone, but the diagrams and tables scroll sideways. Safari or Chrome on a Mac or PC shows it as intended.
The refusal decision
Two thresholds, not chosen by taste. They are read off the ROC curve against asymmetric error budgets — answering a question the corpus cannot support is worse than declining one it can, so the budgets differ. Measured over 715 labelled questions: 396 answerable, 319 unanswerable.
False answers
within budget, interval not yet tight4.7% 95% CI [2.9, 7.6] · 15/319
False refusals
clear of budget2.3% 95% CI [1.2, 4.3] · 9/396
- INSUFFICIENT below
- 0.74
- SUFFICIENT above
- 10.38
- Between
- AMBIGUOUS
refuse — nothing is emitted
answer, every claim cited
answer, flagged partial — never silently upgraded
What is still weak
3 openThese checks run on every push and are expected to fail. They exist to keep known limitations visible rather than letting a green suite imply the problem was solved. A passing probe suite would mean it had stopped looking.
signals_still_overlap
answerable floor 0.49 vs unanswerable ceiling 10.68; the classes overlap, which is why the gate defers most decisions
value_absent_questions_still_score_high
'how long is the observation period between regions' scores 8.85 on retrieval alone; only the value-demand check keeps it out of the sufficient band
chunk_selection_is_weak
the gold chunk is ranked first for 54% of 80 questions; a citation points at a chunk, so this bounds citation precision
Sufficiency classifier
offline artefact · not in the request path
Gradient boosting over 12 retrieval features, trained offline and shipped as ONNX. It ships uncalibrated: Platt scaling collapsed cross-validated AUC from 0.808 to 0.599, and isotonic broke ONNX parity because a step function does not survive float32.
It does not decide anything a visitor sees. Used as a gate it refuses 56.8% of answerable questions against a 10% budget; a two-sided band tuned to fix that measured 4.7% false answers on one pass and 6.1% on held-out folds, over the 5% budget. Choosing the operating point on the same data it is scored against is the defect that produced a “0% false-answer rate” here once already, so the model stays out until it is refitted per fold — ADR-0013.
- Cross-validated AUC
- 0.808
- Training examples
- 715
- Recall at operating point
- 0.366
baseline 0.609 · +0.200
FPR 0.047
Cross-encoder re-ranker
cross-encoder/ms-marco-TinyBERT-L-2-v2 fine-tuned on 1939 pairs. Both metrics ship: document-level was already 0.986 and hid the entire effect — measuring at the level a citation actually points at is what made the improvement visible.
- Chunk MRR, hybrid
- 0.528
- Chunk MRR, fine-tuned
- 0.587
- Re-rank latency p50
- 18.5 ms
before re-ranking
pretrained 0.561
top 20 candidates
Defects
30 logged · 7 severity 1Not one was caught by code review, and not one by a unit test written before it. Every single one was caught by executing something — the assembled system, a measurement over a large labelled set, or a container in CI. That distribution is the finding, more than any individual defect.
| ID | Sev | Defect |
|---|---|---|
| D-001 | 1 | Refusal gate marked 150/265 unanswerable questions answerable — a 56.6% false-answer rate reported at the Sprint 2 gate as zero |
| D-002 | 2 | Evidence gate answered a value-demanding question the corpus never answers, scoring 8.85 |
| D-003 | 3 | Re-ranker experiment returned a null result that was two bugs: a NaN checkpoint and a saturated metric |
| D-004 | 2 | Seed loader destroyed one embedding model's vectors when reloading under another |
| D-005 | 2 | Retry loop re-sent an identical prompt, so it could never succeed |
| D-006 | 1 | Semantic cache served the previous answer to a correction retry (0.886 similarity vs 0.60 threshold) |
| D-007 | 2 | `RUN_BUDGET_USD=0` killed every run mid-stream with an unhandled error |
| D-008 | 2 | CSP blocked React hydration in development; every button was inert HTML |
| D-009 | 1 | The console displayed assessments the governance layer had refused to emit |
| D-010 | 1 | Spend reservation priced against `providers[0]`, under-reserving 4x when failover reached a costlier provider |
| D-011 | 3 | A comment inside a backslash continuation truncated the Semgrep invocation; the scan ran without its exclusion, printed success, and the shell exited 127 on the orphaned flag |
| D-012 | 2 | No body-size limit on the run endpoint; a 200KB body reached the agent and returned 502 instead of being refused |
| D-013 | 1 | The rate-limit pen test could not fail: it sent 8 requests against a limit of 20, and its pass condition accepted `all(c >= 400)`, so a service that was DOWN reported as correctly rate limited |
| D-014 | 2 | Traceability statuses drifted in both directions: 4 rows used statuses the legend never defined (`Done (design)`/`(gate)`/`(decision)`) which the delivery page counted as done, and 1 row sat at `Planned` while its test had passed for four sprints |
| D-015 | 3 | The first README checker could not fail: it searched for each figure as a substring of the whole file, so changing `Commits \ |
| D-016 | 2 | Sprints 6 and 7 were worked and shipped with no planning ceremony and no plan document; the sprint numbers existed only in defect-log entries, and a Sprint 5 retrospective commitment to raise exactly this was never honoured |
| D-017 | 2 | ADR-0003 placed the agent runtime on Hugging Face Docker Spaces "because it is free"; Docker Spaces are PRO-only, so three sprints of deployment work targeted a platform that cannot host it at $0 |
| D-018 | 1 | CI never built the web application. A Dependabot PR taking Next 15→16 reported 10/10 green while the production build failed on all seven pages; no job had ever run `next build` |
| D-019 | 2 | Two governance documents (PROJECT_RECORD.html, SPRINT_08_PLAN.md) asserted the web app was undeployed and Sprint 8 was blocked on credentials for a full week after the real deploy landed; caught only because the Product Owner quoted the stale text back and asked "true?" |
| D-020 | 2 | 20 of 45 `Planned` rows in the traceability matrix (BR-001/004/006/007/008/009/010, FR-002/005/006/007/010/014/015/016/020/025/026/027/029) had real, passing, CI-green tests months before the row was updated — a recurrence of D-014's exact root cause at 20x the scale, caught only by an explicit read-the-code-not-the-doc audit requested by the Product Owner ("I want all of the 45 to be done"). `PROJECT_RECORD.html` (the same document D-019 was found in) independently carried the same stale `13 done` / `45 Planned` figures in three places, hand-typed rather than generated |
| D-021 | 1 | Every scene on the landing page rendered at `opacity: 0` without JavaScript. `Scene` revealed itself with an `IntersectionObserver` and a `visible` state starting `false`, so the server-rendered markup carried an inline `opacity: 0` — measured with JS disabled, all 7 scenes computed to 0. The component's own docstring claimed "content is present in the DOM and legible with animation removed"; it was present and invisible. Reached production and sat there for four sprints |
| D-022 | 3 | The public architecture diagram (`SystemMap.tsx`) labelled the runtime "hugging face space" for a week after ADR-0012 moved it to Northflank — the same stale-claim class as D-017 and D-019, on the surface a reviewer is most likely to read |
| D-023 | 2 | All 12 "read the decision" links on the public architecture surface pointed at the ADR *directory* rather than at a record: the href was built by concatenating a path ending `/adr/` with a filename the derived record never carried. Twelve dead links, each rendering as an ordinary underlined title. Found by the first e2e assertion that every decision link resolves to a `.md` |
| D-024 | 2 | The memory panel rendered the session id, which is read from `sessionStorage` during render — `""` on the server, a real id in the browser. Every page carrying the console threw React #418 (hydration mismatch) in production. Reached `main`; caught by CI, on a build CI made itself |
| D-025 | 2 | The local e2e suite reported green against a STALE build twice: Playwright's `reuseExistingServer: !process.env.CI` reused a server left running from an earlier build, so the run exercised the previous binary. It masked a fix that had not been rebuilt, then masked D-024 entirely — CI caught that one only because it builds from scratch |
| D-026 | 3 | Two labels on the public architecture diagram were struck through by the dashed lines they sat on. The masking plates were sized in SVG user units, so a width that covered the text on macOS was too short on Linux and at mobile scale — the fix for the first version broke on a platform the first version happened to pass on |
| D-027 | 2 | The public architecture diagram asserted three things about the system that were not true. The arrow pointing at **Upstash had no source node** — its tail began in empty space, because the only caller is the edge rate limiter and the node was drawn in the runtime's data column. The node was labelled **`redis · vector`**; Upstash Vector appears exactly once in the repository, in `training/benchmark_vector_store.py`, as the managed-store comparison arm for ADR-0011, and has never served a request. And an edge left the **Provider chain for Neon**, which never opens a database connection — nothing in `router.py` imports the db module. Reported by the Product Owner reading the deployed page |
| D-028 | 3 | The role tabs on the story surface gave **no hover feedback at all once selected**: `.role-tab:hover` and `.role-tab[data-on="true"]` have identical specificity (0,2,0), so the later rule took back both `color` and `background` and the pointer sat over a control that showed no sign of being one. Selection was also carried by **colour alone** — `color`, `background` and `border-color`, every one of which a forced-colors mode overrides — so in Windows High Contrast the chosen role was indistinguishable from the other ten, and the page could not tell you whose story you were reading. The global `:focus-visible` additionally set `border-radius: 4px`, squaring off the 999px pill for keyboard users only. Reported by the Product Owner |
| D-029 | 2 | The label of a role tab was **unreadable for ~60ms every time it was selected**. `transition: color, background` interpolates the two independently, and the path from unselected (#a1a1a8 on #0b0b0d) to selected (#000 on #f5f5f7) runs the text toward black while the background is still near-black. Measured through the fade: contrast 7.66 → 5.39 → **1.71** → 8.62 → 19.29. Reported by the Product Owner as "the button does not display the text that is selected", with a screenshot that had caught the surface mid-fade |
| D-030 | 3 | The traceability matrix cites **26 sprint-story identifiers that no sprint plan ever defines** — S1-DATA, S1-FEED, S1-SEED, S2-CACHE, S2-CHAOS, S2-EVAL, S2-GOV, S2-GUARD, S2-RAG, S2-ROUTER, S2-SPEND, S3-CITE, S3-MEM, S3-RCA, S3-STREAM, S3-TRACE, S3-TRIAGE, S4-PERF, S4-UX, S5-ARCH, S5-DELIV, S6-DELIV, S6-REVIEW, S9-DATA, S9-GOV, S9-KT. Sprints 5 onward list their stories in a table; sprints 1-4 never did, so the matrix's story column points at records that do not exist. Three code comments also cite FR-036, FR-040 and FR-046, none of which appear in the matrix — they are the far ends of range shorthand (`FR-030..FR-036`) written without checking the range existed. Found within minutes of building the lexicon (FR-034), which is the argument for it |
Postmortems
derived at 2026-09-01T02:03:26.660Z from 8d121db