SANDSCOPE / DELIVERY
The record, not the claim
Every number on this page is derived from the repository at build time or read live from the GitHub API. None is typed by hand. The defects are published including the ones that were embarrassing, because a delivery record containing only successes is not evidence of anything.
derived at 2026-09-05T13:25:00.586Z from d66bfb5
Continuous integration, live
reading GitHub…
Every push runs governance gates, lint, strict type checking, the offline suite, integration tests against a real Postgres with pgvector, a statistics suite checked against scikit-learn and scipy, a smoke test of the assembled system, and both evaluation suites. A red pipeline blocks merge.
Scale
- commits
- 10
- test functions
- 423
- lines of code
- 20,574
- lines of documentation
- 5,429
- code to docs
- 3.8:1
- architecture decisions
- 14
across 24 files; pytest reports more because of parametrised expansion
excluding dependencies and generated files
56 documents
Defects
30 found and fixed during the build, 7 of them severity one. Every one is guarded by a regression test. None was found by code review.
| id | sev | what broke | root cause |
|---|---|---|---|
| D-001 | 1 | Refusal gate marked 150/265 unanswerable questions answerable — a 56.6% false-answer rate reported at the Sprint 2 gate as zero | Test set too small and too easy, written by the implementer |
| D-002 | 2 | Evidence gate answered a value-demanding question the corpus never answers, scoring 8.85 | Similarity signals cannot distinguish "about this subject" from "answers this question" |
| D-003 | 3 | Re-ranker experiment returned a null result that was two bugs: a NaN checkpoint and a saturated metric | NaN sorts as a no-op; document-level MRR was already 0.986 |
| D-004 | 2 | Seed loader destroyed one embedding model's vectors when reloading under another | `chunk_embedding` cascades on chunk delete |
| D-005 | 2 | Retry loop re-sent an identical prompt, so it could never succeed | Design oversight: the edge existed, the feedback did not |
| D-006 | 1 | Semantic cache served the previous answer to a correction retry (0.886 similarity vs 0.60 threshold) | **Emergent interaction between two individually correct, individually measured components** |
| D-007 | 2 | `RUN_BUDGET_USD=0` killed every run mid-stream with an unhandled error | Fail-fast not applied to configuration |
| D-008 | 2 | CSP blocked React hydration in development; every button was inert HTML | A policy correct for production made development impossible |
| D-009 | 1 | The console displayed assessments the governance layer had refused to emit | Rendering ignored the run outcome |
| D-010 | 1 | Spend reservation priced against `providers[0]`, under-reserving 4x when failover reached a costlier provider | The guard assumed the cheapest candidate would serve |
| D-011 | 3 | A comment inside a backslash continuation truncated the Semgrep invocation; the scan ran without its exclusion, printed success, and the shell exited 127 on the orphaned flag | Shell semantics: a comment ends a continuation. Every visible signal pointed elsewhere |
| D-012 | 2 | No body-size limit on the run endpoint; a 200KB body reached the agent and returned 502 instead of being refused | Cost bounds were applied to spend and rate, but not to input length |
| D-013 | 1 | The rate-limit pen test could not fail: it sent 8 requests against a limit of 20, and its pass condition accepted `all(c >= 400)`, so a service that was DOWN reported as correctly rate limited | A test written to pass rather than to detect |
| D-014 | 2 | Traceability statuses drifted in both directions: 4 rows used statuses the legend never defined (`Done (design)`/`(gate)`/`(decision)`) which the delivery page counted as done, and 1 row sat at `Planned` while its test had passed for four sprints | A status column maintained by hand, rendered publicly as fact |
| D-015 | 3 | The first README checker could not fail: it searched for each figure as a substring of the whole file, so changing `Commits \ | 54` to 99 still passed because "54" appears in "54% of questions" |
| D-016 | 2 | Sprints 6 and 7 were worked and shipped with no planning ceremony and no plan document; the sprint numbers existed only in defect-log entries, and a Sprint 5 retrospective commitment to raise exactly this was never honoured | Governance applied to the product but not to the process; the control relied on someone remembering |
| D-017 | 2 | ADR-0003 placed the agent runtime on Hugging Face Docker Spaces "because it is free"; Docker Spaces are PRO-only, so three sprints of deployment work targeted a platform that cannot host it at $0 | A pricing claim recorded in an ADR without ever being read from the pricing page |
| D-018 | 1 | CI never built the web application. A Dependabot PR taking Next 15→16 reported 10/10 green while the production build failed on all seven pages; no job had ever run `next build` | A pipeline whose green tick exercises none of the code under change |
| D-019 | 2 | Two governance documents (PROJECT_RECORD.html, SPRINT_08_PLAN.md) asserted the web app was undeployed and Sprint 8 was blocked on credentials for a full week after the real deploy landed; caught only because the Product Owner quoted the stale text back and asked "true?" | The traceability guard catches a requirement claiming Done falsely (overclaiming); nothing caught a document claiming Blocked falsely once it resolved (underclaiming) |
| D-020 | 2 | 20 of 45 `Planned` rows in the traceability matrix (BR-001/004/006/007/008/009/010, FR-002/005/006/007/010/014/015/016/020/025/026/027/029) had real, passing, CI-green tests months before the row was updated — a recurrence of D-014's exact root cause at 20x the scale, caught only by an explicit read-the-code-not-the-doc audit requested by the Product Owner ("I want all of the 45 to be done"). `PROJECT_RECORD.html` (the same document D-019 was found in) independently carried the same stale `13 done` / `45 Planned` figures in three places, hand-typed rather than generated | `check-traceability.mjs` (built for D-014) only fails the build in the overclaiming direction — a `Done` row with no matching test. It has never checked the reverse: a `Planned` row whose named-or-equivalent test already exists and passes, which the public delivery page then undercounts. `PROJECT_RECORD.html` is not generated from `delivery.json` at all, unlike the `/delivery` page and `README.md` — a third recurrence in the same file |
| D-021 | 1 | Every scene on the landing page rendered at `opacity: 0` without JavaScript. `Scene` revealed itself with an `IntersectionObserver` and a `visible` state starting `false`, so the server-rendered markup carried an inline `opacity: 0` — measured with JS disabled, all 7 scenes computed to 0. The component's own docstring claimed "content is present in the DOM and legible with animation removed"; it was present and invisible. Reached production and sat there for four sprints | Motion implemented in JavaScript when the same reveal already existed in CSS (`.reveal`, scroll-driven, `@supports`- and `prefers-reduced-motion`-gated) on another page. A `next build` cannot see it, and there was no browser test that could |
| D-022 | 3 | The public architecture diagram (`SystemMap.tsx`) labelled the runtime "hugging face space" for a week after ADR-0012 moved it to Northflank — the same stale-claim class as D-017 and D-019, on the surface a reviewer is most likely to read | An ADR superseded a platform decision; the prose documents were updated and an inline SVG label was not. `check-deploy-claims.mjs` (built for D-019) greps documents under `docs/`, not components under `apps/web/src` |
| D-023 | 2 | All 12 "read the decision" links on the public architecture surface pointed at the ADR *directory* rather than at a record: the href was built by concatenating a path ending `/adr/` with a filename the derived record never carried. Twelve dead links, each rendering as an ordinary underlined title. Found by the first e2e assertion that every decision link resolves to a `.md` | `derive-surfaces.mjs` omitted the ADR filename, and the page interpolated the missing field into a template literal — where `undefined` would have been visible but an absent property is simply empty string. Nothing checked that a rendered link had a destination |
| D-024 | 2 | The memory panel rendered the session id, which is read from `sessionStorage` during render — `""` on the server, a real id in the browser. Every page carrying the console threw React #418 (hydration mismatch) in production. Reached `main`; caught by CI, on a build CI made itself | A client-only value became RENDERED output. It had been read during render since Sprint 5 and was harmless while nothing displayed it, so the bug was introduced by the component that finally showed it, not by the code that computed it |
| D-025 | 2 | The local e2e suite reported green against a STALE build twice: Playwright's `reuseExistingServer: !process.env.CI` reused a server left running from an earlier build, so the run exercised the previous binary. It masked a fix that had not been rebuilt, then masked D-024 entirely — CI caught that one only because it builds from scratch | Playwright's documented default optimises for iteration speed and assumes the running server matches the working tree. For a suite whose whole purpose is to catch what `next build` cannot see, a local pass that does not exercise the current code is worse than no local run, because it is believed |
| D-026 | 3 | Two labels on the public architecture diagram were struck through by the dashed lines they sat on. The masking plates were sized in SVG user units, so a width that covered the text on macOS was too short on Linux and at mobile scale — the fix for the first version broke on a platform the first version happened to pass on | A mask sized in absolute units cannot track text whose width depends on the font the platform resolved. Only reachable by rendering on more than one platform, which is what the mobile project and Linux CI now do |
| D-027 | 2 | The public architecture diagram asserted three things about the system that were not true. The arrow pointing at **Upstash had no source node** — its tail began in empty space, because the only caller is the edge rate limiter and the node was drawn in the runtime's data column. The node was labelled **`redis · vector`**; Upstash Vector appears exactly once in the repository, in `training/benchmark_vector_store.py`, as the managed-store comparison arm for ADR-0011, and has never served a request. And an edge left the **Provider chain for Neon**, which never opens a database connection — nothing in `router.py` imports the db module. Reported by the Product Owner reading the deployed page | Same class as D-022 and D-019: a diagram is prose that nobody proofreads. The SVG is valid and renders without complaint, so a type check, a build and a screenshot all pass while it is wrong about the system. Nothing had ever asserted that an arrow starts and ends on a box |
| D-028 | 3 | The role tabs on the story surface gave **no hover feedback at all once selected**: `.role-tab:hover` and `.role-tab[data-on="true"]` have identical specificity (0,2,0), so the later rule took back both `color` and `background` and the pointer sat over a control that showed no sign of being one. Selection was also carried by **colour alone** — `color`, `background` and `border-color`, every one of which a forced-colors mode overrides — so in Windows High Contrast the chosen role was indistinguishable from the other ten, and the page could not tell you whose story you were reading. The global `:focus-visible` additionally set `border-radius: 4px`, squaring off the 999px pill for keyboard users only. Reported by the Product Owner | An equal-specificity collision reads as a working rule in review; only the cascade order decides it, and nothing in the suite exercised hover-on-selected. The stylesheet carried no `forced-colors` block anywhere, so every `data-on` control on the site shared the second half of this |
| D-029 | 2 | The label of a role tab was **unreadable for ~60ms every time it was selected**. `transition: color, background` interpolates the two independently, and the path from unselected (#a1a1a8 on #0b0b0d) to selected (#000 on #f5f5f7) runs the text toward black while the background is still near-black. Measured through the fade: contrast 7.66 → 5.39 → **1.71** → 8.62 → 19.29. Reported by the Product Owner as "the button does not display the text that is selected", with a screenshot that had caught the surface mid-fade | Every check anyone would think to run — reading the CSS, reading the cascade, measuring computed style — samples the SETTLED state, and the settled state was always correct at 19.29:1. The defect existed only in the transition between two individually-legible states, which nothing had ever sampled. Two earlier investigations in the same session concluded "renders correctly" for exactly this reason |
| D-030 | 3 | The traceability matrix cites **26 sprint-story identifiers that no sprint plan ever defines** — S1-DATA, S1-FEED, S1-SEED, S2-CACHE, S2-CHAOS, S2-EVAL, S2-GOV, S2-GUARD, S2-RAG, S2-ROUTER, S2-SPEND, S3-CITE, S3-MEM, S3-RCA, S3-STREAM, S3-TRACE, S3-TRIAGE, S4-PERF, S4-UX, S5-ARCH, S5-DELIV, S6-DELIV, S6-REVIEW, S9-DATA, S9-GOV, S9-KT. Sprints 5 onward list their stories in a table; sprints 1-4 never did, so the matrix's story column points at records that do not exist. Three code comments also cite FR-036, FR-040 and FR-046, none of which appear in the matrix — they are the far ends of range shorthand (`FR-030..FR-036`) written without checking the range existed. Found within minutes of building the lexicon (FR-034), which is the argument for it | Same class as D-023's twelve links to a directory: a reference that renders as an ordinary reference and resolves to nothing. Nothing had ever checked that a cited identifier has a record, because until the lexicon existed nothing had ever enumerated the citations |
Requirements
62 requirements, each traced to a story and a named test. A requirement with no test fails the build — the gate refuses to pass rather than warning, so the traceability matrix cannot quietly go stale.
- total
- 62
- delivered
- 62
- planned
- 0
Show all 62 requirements, with the test each one names
| ID | Requirement | Test | Sprint | Status |
|---|---|---|---|---|
| VIS-001 | Reads as production-grade engineering to a senior reviewer | apps/web/e2e/reviewer_journey.spec.ts | 7,9 | done |
| PR-001 | Named-role PDLC with explicit sign-off gates | check-docs.mjs (artifact + gate presence) | 0 | done |
| AC-002 | Every PDLC claim verifiable; no simulated delivery metrics | test_delivery_reads_live_github | 0,5,7 | done |
| AC-001 | Experience layer separated from agent runtime | test_bff_sends_exactly_the_fields_the_runtime_accepts | 0,2,7 | done |
| NFR-001 | Effort directed at delivery, not deliberation | Sprint velocity vs. committed points | 0 | done |
| NFR-002 | Zero infrastructure cost | test_no_paid_service_in_deploy_manifest | 0,6,7 | done |
| NFR-003 | First meaningful paint under 2.5s on cold 4G | apps/web/e2e/performance_budget.spec.ts | 5,9 | done |
| NFR-004 | Public endpoint survives untrusted traffic without unbounded cost | apps/web/src/lib/ratelimit.test.ts | 2,9 | done |
| NFR-005 | Runtime holds no persistent local state | TestNoLocalState::test_serving_requests_writes_nothing_to_the_package_directory | 1,9 | done |
| DR-001 | Visual quality meets apple.com product-page standard | apps/web/e2e/motion_and_reduced_motion.spec.ts | 5,9 | done |
| FR-001 | Product name changeable without code changes | check-config.mjs slug-derivation invariant | 0 | done |
| SD-001 | Deliverable is a working product, not a portfolio listing | PO acceptance at each sprint review | 0 | done |
| SD-002 | Synthetic data only; no real customer data | TestSyntheticOnly::test_no_data_generation_module_reads_from_an_external_source | 1,9 | done |
| SD-003 | No job-application or resume tooling | PRD §8 scope exclusion, reviewed at each gate | 0 | done |
| SD-004 | Domain is agent reliability; incident triage is the workload | PO acceptance | 0 | done |
| CR-001 | Resume content restrictions do not apply | BRD §7 scope statement | 0 | done |
| INF-001 | Containerised runtime, Vercel experience layer | test_container_binds_7860 | 0,6,7,8 | done |
| OPS-001 | Host disk reclamation authorised | Pre-build disk check recorded in sprint log | 0 | done |
| BR-001 | Triage an incident, produce a hypothesis with cited evidence | test_uncited_claims_loop_back_then_escalate | 2,4 | done |
| BR-002 | Demonstrate PDLC/SDLC/CI-CD/Agile/Scrum verifiably | test_delivery_reads_live_github, test_sprint_velocity_is_computed_from_commits_not_typed | 6,9 | done |
| BR-003 | Continue operating when a provider fails or rate-limits | test_workflow_completes_when_first_provider_fails | 2 | done |
| BR-004 | Refuse when evidence does not support an answer | test_no_unanswerable_question_is_ever_marked_sufficient | 2 | done |
| BR-005 | Record a full, inspectable execution trace per run | TestSpans::test_spans_are_persisted_and_readable_after_the_run | 2,4,9 | done |
| BR-006 | Block high-risk actions pending human approval | test_await_approval_has_no_edge_to_a_working_node | 2 | done |
| BR-007 | Attribute tokens and cost to each run | test_actual_cost_is_ledgered_after_the_response | 2 | done |
| BR-008 | Retain session and cross-incident memory | TestMemory::test_recall_is_newest_first_and_capped | 3 | done |
| BR-009 | Evaluate quality against a fixed golden set | test_core_passes | 2 | done |
| BR-010 | Reduce redundant model calls via semantic caching | test_paraphrase_hits_the_semantic_tier | 2 | done |
| BR-011 | Present own architecture, decisions and delivery record | apps/web/e2e/architecture_and_delivery.spec.ts | 6,9 | done |
| FR-002 | Simulated production estate, deterministically seeded | test_same_seed_yields_the_same_incident | 1 | done |
| FR-003 | Incident feed on schedule and on demand | TestIncidentFeed::test_generate_produces_a_fresh_incident_on_demand | 1,9 | done |
| FR-004 | Live triage run streamed to the client | test_node_events_are_emitted_in_the_graphs_actual_topological_order | 3,9 | done |
| FR-005 | Cited evidence panel; unsupported claims marked | test_escalation_emits_no_proposal | 3 | done |
| FR-006 | Explicit refusal on insufficient evidence | test_no_unanswerable_question_is_ever_marked_sufficient | 2 | done |
| FR-007 | Human approval gate; approval is terminal | test_await_approval_has_no_edge_to_a_working_node | 2 | done |
| FR-008 | Session memory, visible to the user | apps/web/e2e/trace_viewer.spec.ts | 3,9 | done |
| FR-009 | Postmortem drafting from a completed run | TestPostmortemCitesOnlyRunEvidence::test_a_citation_the_run_never_made_is_marked_unresolved | 3,9 | done |
| FR-010 | Deterministic router with time-boxed provider disabling | test_disabled_provider_is_reenabled_after_the_ttl | 2 | done |
| FR-011 | Visitor-triggered provider failure injection | TestChaosInjection::test_injection_does_not_leak_into_the_next_run | 2,9 | done |
| FR-012 | Semantic cache with visible hit rate and spend avoided | apps/web/e2e/trace_viewer.spec.ts | 2,9 | done |
| FR-013 | Execution trace viewer | apps/web/e2e/trace_viewer.spec.ts | 3,9 | done |
| FR-014 | Cost attribution and pre-flight spend guard | test_a_live_call_is_refused_with_no_budget_open | 2 | done |
| FR-015 | Evaluation harness over the golden set | test_core_passes | 2 | done |
| FR-016 | Known-limitation probe suite that warns every run | test_probe_warns | 2 | done |
| FR-017 | Cinematic scroll-driven product narrative | apps/web/e2e/scroll_scene_progression.spec.ts | 5,9 | done |
| FR-018 | Reduced-motion and mobile comprehension paths | apps/web/e2e/motion_and_reduced_motion.spec.ts | 5,9 | done |
| FR-019 | Interactive architecture view | apps/web/e2e/architecture_and_delivery.spec.ts | 6,9 | done |
| FR-020 | Live CI/CD status from the GitHub API | test_delivery_reads_live_github | 6 | done |
| FR-021 | Requirements traceability rendered publicly | apps/web/e2e/architecture_and_delivery.spec.ts | 6,9 | done |
| FR-022 | Sprint and velocity record from real commit history | test_sprint_velocity_is_computed_from_commits_not_typed | 6,9 | done |
| FR-023 | Decision records rendered with context and consequences | apps/web/e2e/architecture_and_delivery.spec.ts | 6,9 | done |
| FR-024 | Postmortems for real defects hit during the build | apps/web/e2e/architecture_and_delivery.spec.ts | 7,9 | done |
| FR-025 | Change risk review workload on the same orchestration graph | test_both_workloads_share_one_topology | 5 | done |
| FR-026 | Labelled evaluation dataset, labels true by construction | test_every_answerable_question_names_its_gold_document | 3 | done |
| FR-027 | Statistical evaluation of the refusal gate | test_youden_selects_the_point_furthest_from_the_diagonal | 3 | done |
| FR-028 | Trained, calibrated evidence-sufficiency classifier | TestTheClassifierIsNotInTheLivePath::test_the_evidence_gate_does_not_import_the_classifier | 3,9 | done |
| FR-029 | Cross-encoder re-ranker trained in PyTorch, served via ONNX | test_onnx_matches_torch_on_the_recorded_sample | 3 | done |
| FR-030 | Approximate nearest neighbour benchmark | TestRecallAtK::test_exact_search_against_its_own_ground_truth_is_always_one | 3,9 | done |
| FR-031 | The synthetic dataset published in full | apps/web/e2e/data_and_council.spec.ts, TestDatasetSummaryIsCurrent | 9 | done |
| FR-032 | The governance record published | apps/web/e2e/data_and_council.spec.ts | 9 | done |
| FR-033 | Handover document serving non-technical and architect readers alike | apps/web/e2e/handover.spec.ts | 9 | done |
| FR-034 | Every identifier the project cites is followable to its own record and every place it appears | apps/web/e2e/lookup.spec.ts | 9 | done |
Sprints
- 0Inception1 commit
- 1Foundation0.1.01 commit
- 2Agent Core I0.2.01 commit
- 3Applied ML & Evaluation Science0.3.01 commit
- 4Agent Core II0.4.01 commit
- 5Console0.5.01 commit
- 6Experience0.6.01 commit
- 7Proof Surfaces0.7.01 commit
- 8Hardening & Release0.8.01 commit
Ceremonies map to work sessions rather than calendar days, and that mapping is disclosed here rather than dressed up as a two-week cadence. Faking the calendar would have made every number on this page fiction.
Architecture decisions
14 records. Each is immutable once accepted — a reversal is a new record that supersedes it, never an edit — and each carries what it cost, because a decision log that only records upside is a marketing document.
0001Split the experience layer from the agent runtimeaccepted
Context
The product needs a cinematic web experience and a stateful multi-agent runtime. Every credible implementation of the agent side — LangGraph, BM25, the evaluation harness, the local embedder — exists in Python. Every credible implementation of the experience side — React Server Components, streaming, the motion layer — exists in the JavaScript ecosystem. A single runtime forces one of the two to be reimplemented.
Decision
Two deployable units behind a versioned HTTP contract: a Next.js experience layer acting as a backend-for-frontend, and a FastAPI agent runtime.
Consequences
Positive. Each side uses its native ecosystem. The BFF holds provider secrets so the browser never does. Either side can be replaced independently. The pattern matches how large consumer platforms actually deploy — a JS edge in front of polyglot services. Negative. Two deploy targets, two CI paths, and a network hop on every run. Distributed tracing becomes mandatory rather than optional, because a single request now spans two languages. Reversal condition. If the Python-only components were ever replaced with equivalents in TypeScript, this decision loses its justification and the system should collapse to one runtime. That is a real condition, not a formality.
read the full record →0002The product name is build-time configuration, not sourceaccepted
Context
The Product Owner requires the ability to change the product name late without a refactor. A name hardcoded across components, metadata, manifests and asset filenames turns a rename into a risky find-and-replace across the tree.
Decision
`product.config.json` is the only file in which the product name is authored. Every consumer reads it at build time. `scripts/rename.mjs` performs the rename as a data edit and derives slug, wordmark, repository name and hostnames from it. `scripts/check-config.mjs` fails the build if the derived slug ever drifts from the name.
Consequences
Positive. A rename is one command and a validated data change. The invariant is enforced by a gate rather than by discipline. Negative. No component may write the name as a literal, including in tests and copy. This is a rule people forget, so the secret-scanning pass will be extended to flag hardcoded occurrences once the name is final. Deliberately not automated. The remote repository rename is opt-in (`--apply-remote`). Changing a public URL as a side effect of a build script is not acceptable behaviour for a build script.
read the full record →0003Host the agent runtime on Hugging Face Spaces (free CPU)unknown
Context
NFR-002 requires zero infrastructure cost. Candidates evaluated: Google Cloud Run (generous always-free tier, ~1–3 s cold start, requires a card on file), Render free tier (no card, but sleeps after 15 minutes with a ~50 s cold start), Hugging Face Spaces free CPU (no card, 2 vCPU / 16 GB, sleeps only after prolonged inactivity), and Fly.io (no longer meaningfully free).
Decision
Hugging Face Spaces, Docker SDK, free CPU tier. The Executive Sponsor declined card-backed services outright.
Consequences
Positive. No payment instrument required. 2 vCPU / 16 GB is materially more capacity than most free tiers. The 48-hour idle window is far more forgiving than Render's 15 minutes, and a 6-hourly warm-ping makes sleep effectively unreachable. Negative. The container filesystem is ephemeral, so no state may live on local disk — this drives NFR-005 and forces all state into Postgres and Redis. The image must bind port 7860. Build times are sensitive to image size, which is what rules out `torch` (see ADR-0004). Accepted risk (R-01). A visitor arriving after a long idle period sees a cold start. Mitigated by the warm-ping and by streaming placeholder UI, not eliminated.
read the full record →0004No local neural embedding model in the containeraccepted
Context
Dense retrieval needs embeddings. The default choice, `sentence-transformers`, pulls `torch` — 2–3 GB of image weight. Under ADR-0003 that inflates build time severely and risks the free tier's limits, for a corpus small enough that the quality difference is marginal.
Decision
Dense embeddings come from Gemini `text-embedding-004` (768 dimensions, free tier). The offline fallback is a deterministic zero-dependency embedder carried over from prior work, which projects hashed character n-grams into the same 768 dimensions. Lexical retrieval is BM25 in pure Python and is always available.
Consequences
Positive. The image stays small. Retrieval works with no API key at all, which means tests run offline and deterministically. The fallback is a genuine degradation path rather than a hypothetical one. Negative. Dense retrieval depends on an external provider during normal operation, so the embedding provider joins the router's failure surface. Important. The fallback embedder does not produce vectors comparable with Gemini's. See ADR-0005 — this is the constraint that decision exists to enforce.
read the full record →0005Vectors from different embedding models are never comparedaccepted
Context
ADR-0004 introduces two embedding sources. Storing both in one `vector` column and comparing them would produce a cosine similarity that is arithmetically valid and semantically meaningless. This class of bug is silent: retrieval keeps returning results, they are simply the wrong ones, and no error is ever raised.
Decision
Embeddings live in `chunk_embedding(chunk_id, model, dim, vec)` with a unique constraint on `(chunk_id, model)`. Every query filters on the active embedding model. The semantic cache key includes `embedding_model`. When the active model has no stored vectors, retrieval degrades to BM25-only and labels itself degraded — it never falls back to a different space.
Consequences
Positive. Cross-space comparison becomes structurally impossible rather than merely discouraged. Adding a third embedding model later requires no migration. Degradation is explicit and visible to the user. Negative. Storage is duplicated per model. Backfilling a new model is an explicit job. A cache miss occurs whenever the active embedding model changes, which is correct but costs a warm-up. Verification. A regression test asserts that a query under model A never returns rows embedded under model B.
read the full record →0006Human-approval nodes are terminal by constructionaccepted
Context
In prior work on a multi-agent system, a human-approval node had an outgoing edge into a registration node. The graph therefore proceeded to register the item regardless of what the reviewer actually answered. The approval step existed, rendered, collected a decision, and controlled nothing. It read as governance and functioned as a formality — the worst possible failure mode for a control, because it consumes the reviewer's attention while providing no protection.
Decision
`await_approval` has no outgoing edge. Reaching it ends the run. A human decision creates a new run that carries the approval record as input.
Consequences
Positive. Auto-proceed is impossible by graph topology, not by careful coding. The approval record is a first-class row with identity and timestamp, satisfying Sofia's audit requirement. Continuation runs are independently traceable. Negative. Two runs per approved incident instead of one; the UI must join them into a single narrative for the user. Verification. A regression test enumerates the compiled graph's edges and fails if any edge originates at `await_approval`. The test was written to fail against the defective topology before the correct one existed.
read the full record →0007Rate limiting and the spend guard fail closedaccepted
Context
The service is publicly reachable and calls paid-capable model providers. The common default — allow the request when the rate-limit store is unreachable — is chosen so that an infrastructure blip does not degrade user experience. Here that default converts a Redis outage into an unbounded-cost incident, and the endpoint into free LLM capacity for whoever finds it.
Decision
When Redis is unreachable, rate limiting denies. When no budget is open, the spend guard refuses the live model call. Every call is priced at worst case before it is issued; actual cost is written to the ledger afterwards.
Consequences
Positive. The worst outcome of a dependency failure is unavailability, not unbounded spend. Cost is bounded before it is incurred rather than reconciled after. Negative. A Redis outage takes the demo down. Given NFR-002 and a demo workload, that is the correct trade; it would not be for a revenue-bearing service, and that difference is the point of recording it here. Verification. Unit tests simulate an unreachable limiter and assert denial, and assert that a live call with no open budget raises rather than proceeds.
read the full record →0008Similarity thresholds belong to the embedder, not to the cacheaccepted
Context
The semantic cache was first written with a module-level `SIMILARITY_THRESHOLD = 0.86`, chosen because 0.86 is the sort of number that appears in semantic-cache examples. Measuring it over real corpus questions showed the number was wrong in both directions at once. With the offline hashed embedder, paraphrases of the same operational question scored 0.679 to 0.959 and distinct questions scored -0.065 to 0.034. A threshold of 0.86 would have missed genuine paraphrases near the bottom of that range, while sitting far above where the actual decision boundary is. The more consequential finding is why a single constant cannot be correct. A sparse hashed space and a dense neural space have entirely different similarity distributions. Unrelated text scores near zero in the first and routinely above 0.5 in the second. A threshold calibrated in one is not merely suboptimal in the other — in the dense space it would treat unrelated questions as the same question and return the wrong cached answer, with no error raised anywhere. This is the same failure ADR-0005 prevents for the vectors, arriving through the threshold instead.
Decision
`similarity_threshold` is a property of the `Embedder`. `SemanticCache` reads it from whichever embedder it holds. There is no module-level default. `HashingEmbedder.similarity_threshold = 0.60`, derived from the measurement above: an order of magnitude above the highest observed distinct pair, and below the lowest observed paraphrase. The value is biased toward the miss. A missed cache hit costs one model call. A false hit returns a wrong answer to a user and looks exactly like a correct one. Any embedder added later must state its own measured threshold. A test re-derives the separation and fails if it stops holding.
Consequences
Positive. Adding an embedder forces the question "what is the boundary in *your* space" to be answered with a measurement. The regression test makes a silent drift in the embedder visible as a failure rather than as slightly wrong answers. Negative. Every embedder implementation carries a calibration obligation. That is the intended cost. Watch item. The hosted embedder is not yet implemented. It must not inherit 0.60. Its threshold is measured against the same paraphrase and distinct pairs before it is enabled, and the test that does so is a release blocker for it.
read the full record →0009Models are trained offline and served without their training frameworkaccepted
Context
Sprint 3 introduces trained models. ADR-0004 excluded `torch` from the runtime image because it is 2–3 GB, which breaks build times on the free tier and violates NFR-002. That reasoning still holds and is not being reversed. But ADR-0004 was about the runtime image, and it was quietly read as "no machine learning in this project" — which is not what it says, and is a worse constraint than the one that was actually decided. Training does not happen in the serving container.
Decision
Training runs offline, on a development machine, with whatever framework suits the model. Serving runs in the container with the smallest artefact that reproduces the trained behaviour. | Model | Trained with | Served with | Runtime cost | |---|---|---|---| | Evidence-sufficiency classifier | scikit-learn | Exported coefficients / a small pickled estimator | negligible | | Cross-encoder re-ranker | PyTorch | ONNX Runtime | ~50 MB | | Telemetry anomaly detector (if built) | TensorFlow or scikit-learn | ONNX Runtime | shared | Every trained artefact is versioned, checksummed, and accompanied by the evaluation that justified shipping it. A model in the repository without its evaluation is not deployable.
Consequences
Positive. The runtime image stays small and ADR-0003's constraints hold. The split is the standard production pattern rather than a workaround, so it survives scrutiny. Training frameworks never become a deployment dependency, and a framework can be swapped without touching the serving path. Negative. An export step exists between training and serving, and export is where numerical behaviour can silently change. A test asserts that the ONNX output matches the PyTorch output within tolerance on a fixed sample; without it, "the model is deployed" and "the trained model is deployed" are different claims that look identical. Also negative. Training is not reproducible in CI, because CI will not carry the training frameworks. Training scripts pin their seeds and record their environment, and the artefact's checksum is what CI verifies.
read the full record →0010Training labels are true by construction, never model-generatedaccepted
Context
FR-028 needs a labelled dataset: questions paired with whether the corpus can answer them. The existing set is 22 questions written by hand, which is enough to falsify a threshold and nowhere near enough to fit a model. The obvious way to scale it is to have an LLM generate questions and label them. It is fast, it produces plausible data, and it is invalid here. A classifier trained on LLM-assigned labels learns the LLM's judgement of what is answerable, including its errors. Evaluating that classifier against LLM-derived ground truth then measures agreement with the LLM rather than correctness, and reports a high number either way. This product's central claim is that it can tell grounded from ungrounded. A training set whose ground truth is itself a model's opinion would make that claim unfalsifiable.
Decision
Every label is derived from the construction of the example, not asserted about it afterwards. Answerable examples are generated *from* a specific chunk. The chunk is selected first, the question is generated from its content, and that chunk is the gold reference. Answerability is a property of how the example was built. Unanswerable examples are generated by two mechanisms with the same property: 1. Gap topics. Questions about subjects `corpus/GAPS.md` records as absent. Absence is verified by search over the corpus, not assumed. 2. Entity perturbation. An answerable question with its entity replaced by one the corpus does not document in that respect — asking for the `sessions-cache` pool ceiling when only `orders-db` states one. This produces the hard class: full vocabulary overlap, no answer present. An LLM may be used to vary *phrasing*. It is never used to decide a label. The distinction is enforceable: label assignment happens in code that never calls a model, and a test asserts the generation path makes no model call.
Consequences
Positive. Ground truth is auditable — for any example, the reason for its label is mechanical and inspectable. Evaluation measures correctness rather than agreement with a model. The entity-perturbation mechanism produces exactly the failure class that defeated the heuristic gate in Sprint 2, so the hard cases are represented in proportion rather than by luck. Negative. The generated questions are less linguistically varied than a model would produce, so the classifier may be less robust to phrasings that never appear in training. That is a real limitation, it is stated in the model card, and it is measured by holding out a small hand-written set that no generator produced. Splitting. The train/test split is by document, not by question. Two questions generated from the same chunk share nearly all their features; putting one in train and one in test inflates every score and the inflation is invisible.
read the full record →0011No approximate-nearest-neighbour index until the corpus reaches ~5,000 chunksaccepted
Context
The schema defines an HNSW index on `chunk_embedding`. It was added because a vector column with an HNSW index is what a pgvector schema looks like, which is not a reason. This system's corpus is 87 chunks. Whether an index helps at that size is a measurable question and it had not been measured.
Decision
Exact search until the corpus reaches roughly 5,000 chunks. The HNSW index declared in migration 0001 is retained — it costs 360 KB and nothing at query time at this size — but the system does not depend on it and the decision to rely on one is deferred until the measurement above says it pays. Revisit when the corpus exceeds 5,000 chunks, or when exact-search p50 exceeds 20 ms, whichever comes first.
Consequences
Positive. The system uses the simplest thing that works, and there is now a number attached to the word "works" and a threshold at which to revisit. Retrieval stays exact, so ADR-0005's degradation path has no approximation interacting with it. Negative. A corpus that grows past 5,000 chunks without anyone re-running the benchmark will get slow before it gets fixed. The revisit trigger is written above rather than left to be noticed.
read the full record →0012The agent runtime runs on Northflank, not Hugging Face Spacesaccepted
Context
ADR-0003 placed the agent runtime on Hugging Face Spaces using the Docker SDK, on the stated grounds that it was free. That was true when the decision was written and is no longer true: Hugging Face now requires a PRO subscription for Docker and Gradio Spaces, and only Static Spaces remain free. A Static Space cannot run FastAPI, so the decision did not merely become expensive — it became impossible at $0. The failure here was not choosing Spaces. It was never verifying the free tier before building three sprints of deployment work on top of it, and then writing an ADR that recorded the choice without recording how the pricing claim had been checked. It had not been checked at all. The whole point of NFR-002 is that zero infrastructure cost is a requirement with a test behind it; the test asserted that no *paid host* appeared in a manifest, which a host that silently moves behind a paywall passes trivially.
Decision
Run the agent runtime on Northflank, Developer Sandbox plan, in `europe-west` (London).
Consequences
- No SLA, and Northflank documents the Sandbox as not for production. For a demonstration surface that is acceptable and is stated on the surface itself rather than hidden. - A payment method is required on file, on every Northflank plan, even the free one. It is a verification measure and nothing is charged inside the allowance, but it is not a card-free option and should not be described as one. - Two free services, one addon. The architecture fits, with nothing spare. - The image has still never been built locally — no Docker on the development machine. Northflank's first build is the first real build.
read the full record →0013The trained classifier stays out of the live evidence gateaccepted
Context
Sprint 3 trained an evidence-sufficiency classifier: gradient boosting over 12 retrieval features, 715 labelled examples, cross-validated AUC 0.808 against a 0.609 baseline, exported to ONNX and served without scikit-learn (ADR-0009). It has its own unit tests, its own ONNX-parity test, and a panel on `/reliability`. It is not in the request path. `retrieval/evidence.py` decides with a hand-derived three-band heuristic, and has since Sprint 2. A Sprint 9 audit flagged this: three applied-ML requirements (FR-028 classifier, FR-029 re-ranker, FR-030 ANN benchmark) are each genuinely built and rigorously tested as standalone artefacts, and none of the three is wired into what the API actually serves. For a page whose entire argument is that its claims are checkable, a component presented as production behaviour that no request ever touches is the exact failure this project exists to avoid. The obvious remedy is to wire it in. That was measured before it was done.
Decision
The classifier stays out of the live evidence gate. The reliability surface states plainly that it is an offline artefact rather than implying it serves traffic. The measurement is `training/evaluate_classifier_as_gate.py`, re-runnable, and its result is the entire reason for this record.
Consequences
Positive. The published 4.7% false-answer rate continues to describe what the service actually does. The reliability surface stops implying that a model serves traffic when it does not. The measurement is committed and re-runnable, so this decision can be overturned by evidence rather than by opinion. Negative. The gate still defers 87.3% of questions to AMBIGUOUS, which is a real weakness and remains published as one. The classifier remains built, tested, and unused in production — that is honest, and it is still an asset sitting idle. What would change this. The model was fitted on all 715 questions, so it leaks into every fold above; the folds remove only the larger and entirely avoidable leak of selecting the band on its own evaluation data. A clean answer needs the model refitted inside each fold, which needs the training extra (~2GB, ADR-0009) that is deliberately absent from the serving image. Refitting per fold, or labelling more questions, is the path to revisiting this — not a better sweep. Verification. `training/evaluate_classifier_as_gate.py` reproduces every number here. `test_the_classifier_is_not_wired_into_the_live_gate` asserts the decision holds, so the wiring cannot reappear silently in a later refactor.
read the full record →0014One Vercel project, connected to gitaccepted
Context
Two Vercel projects existed for one application, and the arrangement hid a staleness failure for eight days. | | `sandscope` | `sandscope-web` | |---|---|---| | Git connected | github:224Sand/sandscope @main | not connected | | Deploys on push | yes | no — manual CLI only | | Public alias | `sandscope-five.vercel.app` | `sandscope-web.vercel.app` | | `ssoProtection` | `all_except_custom_domains` | `null` | `sandscope-web` is the URL that was shared, and it only updated when somebody ran `vercel --prod` by hand. It therefore went eight days stale carrying a completed sprint — 45 requirements still shown as Planned after an audit had moved the real number — and nobody noticed, because `sandscope` was going green on every push the whole time. A diagnosis error is recorded here because it nearly reversed this decision. The first investigation tested `https://sandscope-l0m4blqhm-sand224ai-8475s-projects.vercel.app`, received a 302, and concluded `sandscope` was behind a login wall and unusable. That is a per-deploy hash URL, and `all_except_custom_domains` applies to exactly those while exempting the clean alias. The alias was never tested. One URL's status was generalised to a whole project, and a plan was written on it that would have deleted the only project that auto-deploys. The Product Owner caught it by pointing out that `sandscope-five.vercel.app` works.
Decision
One project, connected to the repository: `sandscope`. The `sandscope-web.vercel.app` alias moves onto it, so the URL everyone has keeps working and starts updating itself. Consolidating the other way — connecting `sandscope-web` — was rejected: it requires a dashboard OAuth grant, while moving an alias is an API call, and it would mean teaching the manual project a trick the other one already does.
Consequences
Positive. The public site cannot silently go stale; a push is a deploy. One project to look at rather than two, one of which was a decoy. No manual step to forget. Negative. A bad commit reaches production without a human gate. That is bounded by CI — a red pipeline blocks merge, and merge is what deploys — and by `check-deploy-freshness.mjs`, but it is a real reduction in ceremony and worth naming rather than glossing. `sandscope-five.vercel.app` remains as a second alias on the same project. Harmless — it serves the identical deployment — and removing it would break any link already shared. What would change this. If production ever needs a human release gate, the answer is a protected branch or a promotion step, not a disconnected project that someone remembers to deploy.
read the full record →