From 82% to 95% on Existing Hardware — Assessing and Improving an On-Premise AI Assistant

1. Executive Summary

RCTK was tasked with assessing and improving a previously built on-premise retrieval-augmented assistant: 82.2% answerable accuracy against an 85% target, zero answers safe to auto-accept, and an imposed ban on non-self-hosted third-party LLMs. The engagement ran as three phased stages, each closing at a decision gate with standing pause-and-exit rights, with nine predictions registered before any pilot data existed.

The assessment localized the failure to a single stage (delivery of benefit tables to the model) and ruled out the obvious capacity fixes with direct measurements before any improvement budget was spent. The improvement program rebuilt retrieval around a structured key and table-aware ingestion, bought model capacity only after delivery was fixed, and added a deterministic typed-extraction spine that auto-accepts single-cell answers at zero critical errors by construction. On the same 200-question blind set, against the same automated scorer that measured the baseline, the integrated system reached 95.0% answerable accuracy at the peak validated build (171/180, 95% CI [90.8%, 97.3%]), with the entire +12.8-point lift attributable to retrieval plus model capacity, on the same single GPU. At the second gate a balanced configuration (94.4% accuracy / 95.6% automated faithfulness) was selected over a higher-accuracy build, both positions recorded; a Phase-3 refinement then lifted the deployed build to 94.8% / 95.5%.

The remainder of Part II: methodology (§2), the baseline (§3), the assessment (§4), the improvement program (§5), the faithfulness trade-off and the election (§6), productionization and operations (§7), gate and prediction outcomes (§8), production results (§9), the evidence index (§10), and two appendices (the prediction protocol; the deployed stack reference).

2. Engagement and Evaluation Methodology

2.1 The assessment-and-improvement shape

This was not a greenfield build. A functioning, competently engineered system existed with a measurable baseline, so the engagement’s first phase was an assessment of that system: reproduce its reported numbers independently, localize the failure, test the candidate fixes directly enough to keep or discard each one, and validate an improvement path component by component. The improvement phase then implemented the validated path with each intervention measured before the next was started, so the final lift could be attributed to its components rather than asserted (§5.7). The productionization phase transferred the result to the corpus under acceptance criteria fixed in advance. The fallback was set: if the assessment had found gaps too large to close before the deadline, the engagement would close cleanly at the first gate as an audit.

2.2 Decision-gate governance and the conditions discipline

Each phase ended at a decision gate where the choice was proceed, pause, or exit. Gate deliverables were written as pre-reads ahead of each meeting; bad news was in writing before the gate, not surfaced first in the meeting (the faithfulness trade-off memo of §6 and the production incident of §7 are the two worked examples). The gates produced approval conditions (13 at DG1, 16 at DG2), which tracked to closure across the engagement (§8). By the third gate, the pre-read could state plainly that nothing presented would be new.

The conditions followed one register discipline across all three gates: each carries a verifiable criterion and a due gate; each is discharged in evidence or explicitly carried forward; and the register is reviewed at every subsequent gate. The same protection ran in both directions: success criteria moved only by approved change request. A status cadence carried the discipline between gates; the engagement’s one production incident reached that period’s status report, not at the next gate (§7).

Incidents ran under the same written discipline: a severity model with agreed service-level windows for acknowledgment, plan, and resolution; the operating rule of classifying fast on the worst plausible read and then re-classifying with written reasons as facts arrive; and routing with compliance pulled in parallel only when an incident actually has a compliance dimension.

2.3 The pre-registered prediction protocol

Predictions lived in two tiers. Committed meant high diagnostic confidence with the fix well understood: a miss is a serious, reportable event carrying a root-cause analysis. Stretch meant residual uncertainty, usually because a fix was validated but not yet built at corpus scale or depended on an unconfirmed corpus characteristic: a miss is informational. Nine predictions were registered at the Phase-2 kickoff, before any pilot data existed, to be scored at the gates pass or fail and reported regardless of outcome.

The protocol’s teeth are in selective promotion: a number moves from stretch to committed only when its gating evidence arrives. At the kickoff, exactly one prediction was promoted (answer accuracy, on the strength of a duplication check run on the library), while auto-accept and time-savings were held to their own Week-1 evidence. The auto-accept stretch was subsequently held, in sequence, at the first gate, the kickoff, the Week-1 check, the mid-pilot review, and the second gate — five holds in all — before graduating to production on a verification record (§8; Appendix A gives the worked example). The full scorecard, including the one prediction that finished as a held property rather than a clean pass, is in §8.

What keeps the registration from being a list of goals is two mechanics. Commitments are thresholds with margin, not point estimates: accuracy was committed at ≥85% while the expected landing point was around 90–91%, so the commitment could survive ordinary measurement noise without putting the threshold at risk. The downside was owned in advance: a committed miss is reported as a serious event with a root-cause analysis, with no after-the-fact reclassification. And every held prediction carries a stated dependency (auto-accept was held on scanned-table ingestion fidelity, time-savings on the measured search-versus-read split), so a “stretch” label points at a specific open variable instead of hedging by reflex.

2.4 Evaluation methodology and denominators

All pilot-stage numbers in this report come from one instrument: a 200-question blind test set built from call patterns, finalized before any pipeline tuning, and excluded from all training data. Its composition: 180 answerable questions across four categories (benefit lookup 122, eligibility 31, plan-rule interpretation 17, claims 10), each with a gold answer and gold citation, spanning six employer groups so that access-control-scoped retrieval is exercised; difficulty bands of 67 / 41 / 72 (levels 1–3); and a deliberately hard 40-question edge-case tier inside the 180. The remaining 20 questions are adversarial, constructed to be refused: plans outside the asker’s service scope, false premises, questions whose answer is contradicted across superseded documents, and framings that bait a confident answer from a passage that does not support it.

Denominators, used consistently throughout both parts: “answerable accuracy” is scored on the 180 answerable questions only. Refusal behavior on the 20 adversarial questions is reported separately and never blended into accuracy — baseline 90.0%, improved 95.0% adversarial refusal, and zero of the twenty ever routed to the auto-accept path at any operating point.

Scoring used the identical automated scorer before and after every change: bidirectional NLI entailment plus numeric match for value answers scores accuracy; NLI adherence ≥ 0.70 of the answer against its cited passage scores faithfulness; asserting a specific fact no retrieved passage supports scores hallucination. This design does two jobs at once. Because the set and scorer never changed, every before/after delta is apples-to-apples even where the scorer’s absolute level carries bias. And the scorer’s one known artifact, under-crediting a correct table value restated in prose, was identified, diagnosed, and handled in the open: it is the subject of §6, and a human re-rating was retained alongside the automated number for that reason. The live measurements (time savings, adoption, production accuracy) belong to Phase 3 and the support cycle (§§7–9).

One more note about the instrument belongs here, because every absolute number in this report inherits it: an automated NLI scorer is known to over-credit short or generic answers, so absolute levels carry an optimism bias even where relative comparisons are reliable. The engagement’s design leans on the relative reading (identical set, identical scorer, before and after) and pre-registered a human re-rating protocol for the one metric, citation faithfulness, where the automated instrument’s specific weakness mattered most (§6).

2.5 The synthetic corpus and the scale caveat

Improvement experiments ran on a synthetic, production-faithful replica of the corpus — roughly 8,000 documents scaled up from the ~1,200-document live inventory, deliberately over-duplicated (a median of roughly fifty value-similar sibling documents per benefit key) so the delivery problem was stressed harder than production would stress it. The replica reproduced the baseline’s metrics exactly, which validates the measurement harness; it was not claimed to validate corpus structure. The two characteristics a replica cannot establish (how heavily the library duplicates documents, and whether scanned-table values survive text extraction) were flagged as the engagement’s #1 caveat at the first gate and retired in stages: a pre-signature duplication sample across 32 stratified benefit keys (finding essentially one in-force source-of-record document per key), the Week-1 corpus-wide census that confirmed it, and the Phase-3 validating run on the production library itself (§7), which reported 91.7% raw and 94.2% / 95.4% after the scanned stock was recovered. Every pilot number in §§3–6 should be read with that pedigree in mind: measured on the replica, transferred to production under a gate designed for the purpose. Where the replica’s duplication artifact binds a metric, the engagement reported paired readings, production-faithful and replica-pessimistic; §4.4 and §5.1 carry the worked examples.

3. The Baseline System (the “Before”)

3.1 Architecture as built

3.2 The baseline scorecard

Independently reproduced on the 200-question blind set, matching the reported figures exactly and validating both the measurement harness and the team’s measurement discipline. Five of eight gate criteria passed:

3.3 Where it failed, precisely

The category shape (F2, Part I) is the diagnosis in miniature: one category, two-thirds of the set, carried essentially the whole gap. By difficulty, the middle band was weakest (level 1: 88.1%; level 2: 73.2%; level 3: 81.9%). The 32 failures decompose into 22 retrieval-bounded (the gold document never reached the delivered top-10), 5 model false-refusals, and 5 wrong-number extractions. Conditional accuracy: 90.8% when the gold document was retrieved, 69.0% when it was not. The reader was capable; delivery was the ceiling. Overall, the correct source document reached the model only about six times in ten, confirming the pre-engagement estimate (the instrumented per-category figures are in the table above). One scoring note: the recall column above is the baseline harness’s own document-match metric, quoted because it is the instrument used; the assessment’s stricter exact-source and looser family-level forensics in §5.1 are different instruments and read lower and higher respectively.

Operationally, mean latency was 20.6 s (P95 37.1 s): most likely marginal for a specialist on a live call. And the confidence model could not rescue trust: at test AUC 0.667, it auto-accepted nothing at any threshold, because a confidence score bolted onto a free-text answer cannot accept answers the system gets wrong.

The verdict the assessment delivered, and this report stands behind: a defensible internal build (strong citations, perfect isolation, honest refusals, clean PII hygiene) with one broken stage, which is what made a targeted engagement viable at all.

4. The Assessment (Phase 1)

4.1 Reproduce first

The assessment’s first act was to re-run the 200-question evaluation through the system with an independent harness, scoring with the same NLI methodology. It landed on 82.2% answerable, the reported figure to the decimal. That mattered methodologically (every later before/after stands on this harness) and organizationally.

4.2 The root-cause chain

1. The failure was categorical, not general. Benefit lookups: 76.2% (n=122). Every other category: ≥ 93.5% (§3.3).
2. It was not a plan-selection failure. Plan and plan-year filtering measured essentially perfect — 0% wrong-plan on the 60-question localization sample (at most ~6% at the 95% upper bound). Of the localization misses, 76% were right-plan, right-year, wrong-document-or-section; a manual classification attributed ~78% of accuracy errors to tabular value-lookups (an annotation, corroborated by the instrumented category and delivery measurements).
3. The diagnosis number: gold table-chunk recall@10 under the baseline system’s chunking was 3.6% on the numeric value-lookup slice: the correct table chunk almost never reached the model’s context. (The report uses several distinct recall metrics; each appears only under its exact name.)
4. Delivery, not comprehension. Conditional accuracy was 90.8% with the gold document retrieved versus 69.0% without it; handed the exact correct table, the existing 8B model read the right value ~88% of the time.
5. What did arrive was often corrupted. 30.6% of table-bearing chunks merged rows from two or more distinct tables, and the chunker’s 50-token overlap prepended the previous table’s tail to each table chunk — cross-table numeric contamination even when something relevant was delivered (quantified in full by the Phase-2 index forensics, §5.1).
6. The structural conclusion. Benefit values are grids of near-identical numbers across hundreds of look-alike same-plan tables. No text-similarity ranking (dense, lexical, cross-encoder, or fact-ized) can isolate the one authoritative cell. That is not a problem more scale can fix; it requires structure.

4.3 Ruled out, with data

The four places an improvement budget would naturally have gone were each tested directly, on the live system, before Phase 2 was scoped:

Cheaper text-based delivery patches measured the same way: metadata-targeted table delivery 21%, section/parent-table expansion 36% — directionally better, capped by document recall. One inherited null was a correction: the chunking was already heading-aware and table-preserving within its token budget (better than the root-cause analysis described), so “fix the chunking” alone was never the lever either. The residual was table flattening on roughly 9% of plan documents, a contributor but not the cause.

Two of these nulls were re-tested later: the embedder null was re-confirmed on the improved index under fair, instruction-prompted conditions (§5.1), and the capacity null, measured on broken delivery, was re-opened from scratch once delivery was fixed — where it reversed (§5.2). A null is a measurement of a system state, not a permanent truth.

4.4 The validated path — component by component, then assembled

• Locate. A structured key (employer, plan, year, benefit line) plus document precedence located the authoritative source-of-record document at 88–93% precedence-recall, where text similarity had managed 3.6–21%. Structure solved the locate problem no similarity method could.
• Read. Typed extraction — select the cell coordinate, read the value verbatim in code, abstain on ambiguity. Integrated with no oracle on the full 43-question tabular census, structured delivery scored 83.7% [70%, 92%] against the 46.5% identical-generator baseline, lifting the hard numeric core from 16% to 72%, with faithfulness up and zero hallucination. For auto-accept, generate-then-ground reached 56% coverage at ~88% precision but took 3 critical errors; inverting the step (select → read verbatim → abstain) measured 26% coverage at 100% precision and zero critical errors, model-free. That zero is a property of the construction, the gate cannot accept what it had to guess at — which is why the assessment registered the coverage number as a stretch rather than a commitment.
• Assemble. With a question router over the whole 200-question set, the prototype measured 90.6% answerable [85%, 94%] at the F1-optimal operating point on the production-faithful reading. The assessment refused to promise that number: the replica could not settle how heavily the library duplicates plan documents, so the honest range was 83–91%, flagged as the engagement’s #1 caveat, and the first week of Phase 2 was designed to measure the deciding variable on the corpus before any build budget was consumed. The gate record carries the phrase “validated, not yet built.”

The assessment also stress-tested its own candidate component, and disclosed the three places the bare deterministic gate broke. Heavy query abbreviation produced 2 of 11 confidently wrong reads, closed by query normalization (11 of 11 correct after). Light OCR character corruption in scanned tables produced 2 of 11 confidently wrong reads: a corrupted value can still look type-valid, so ingestion fidelity, not the in-system type check, was identified as the binding control and became a Week-1 gate item (and, eventually, §5.5). Disagreeing superseded or duplicate tables were picked silently by the bare gate (0 of 3 safe); a routed model backstop measured 3 of 3 safe abstentions, with precedence and provenance at delivery adopted as the primary mitigation. Each failure mode and its mitigation went into the gate deliverable; the bare component was never proposed for deployment alone. And on the question class that had broken the pilot (coordination of benefits across two documents), the validated path computed what was genuinely arithmetic (3 of 3) and safely abstained where the answer was rule-dependent (3 of 3), at zero critical errors — made safe, not automated, and the assessment said which.

The #1 caveat was given teeth in two ways. The Week-1 check was delivered as a pre-built instrument, demonstrated on the replica (where it correctly detects both the duplication and the OCR corruption it exists to find), so Week 1 would supply a reading on the corpus rather than a methods debate. And the downside was priced before the gate: if the validation failed, the choice could be made to de-scope to the committed tier, or exit at work performed to date. That fallback scenario (roughly the baseline’s accuracy, with the safety, corpus, and verification-tax work intact but not the accuracy gain) was written into the assessment itself, so an actual executive conversation would have been early and informed.

4.5 The surfaced finding

The legacy repository. The corpus audit asked the inverted question (is everything a specialist would actually use wired into the assistant?) and surfaced a legacy document repository on a retired file server, left behind by a prior migration: roughly 600 retired plan documents, invisible to the assistant, and to its access controls. Classified as a non-reportable internal control finding; scoped at the gate as a change order importing the corpus under the same audit envelope as everything else. The discovery doubled as a root-cause insight: the documents governing the hardest questions were exactly the ones the system could not see.

The gate that closed the phase is narrated in Part I and scored in §8.

5. The Improvement Program

Seven engineering stages, each measured before the next began, all on the instruments of §2.4. This section compresses each stage to its findings and decisions; the stage designs, sweeps, and per-question diagnostics live in the improvement evaluation report and the seven stage reports indexed in §10.

5.1 Retrieval — structured delivery (IV-2)

Forensics on the baseline index first: across the 200-question set, gold-document recall@10 was 28.3% (exact source-of-record) and gold-family recall@10 75.6%, with gold-table recall@10 on the 43 table questions at 18.6%; 24.1% of chunks contained a table, 30.6% of those merged two or more distinct tables, 39.2% mixed table with prose, and the 50-token overlap bled each previous table’s tail into the next chunk.

The fix had two parts working together. Table-aware, document-type-specific chunking: every table becomes its own chunk (never split, never merged, zero overlap), prefixed with a structured descriptor (section title + column headers + row labels) so a grid of numbers becomes retrievable by what it answers; noise-bearing and sensitive-content tables are flagged out of value retrieval. On the full corpus this produced 10,969 isolated, descriptor-tagged benefit tables and took merged-table contamination from 30.6% to 0.0%, with the gold table present in its own document’s chunks 100% of the time. A two-stage structured retriever: stage one resolves (employer, plan, year, benefit line) by metadata precedence to the single source-of-record document, a filter rather than a similarity search; stage two ranks that document’s table chunks by descriptor and delivers the top two whole.

The ablation row is the order-of-operations finding: clean chunks are necessary but similarity still cannot rank the right table among near-duplicates; the structured key is the lever. The remaining knobs were re-confirmed null on the new index: the fusion-weight sweep peaked at 25.6% gold-table@10, the accept threshold had zero effect, the cross-encoder reranker added +2.3 pp, and the 4B embedder on a fair, instruction-prompted re-run scored 88.4%@1 against the 0.6B’s 93.0%@1 while costing ~6 GB of VRAM, so the existing embedder was retained. A confirmatory generation spot-check closed the loop: delivered context took the tabular slice from 18% to 83.3% (35/42), within 2.4 pp of the 85.7% hand-the-gold-table ceiling. Delivery recovered essentially the full prize.

Two residuals were carried forward. The replica’s exact-source-document recall reads 2.8% under precedence, a duplicate-family artifact of the synthetic corpus (on the one-source-per-key library, resolving the family is resolving the document; the Week-1 census confirmed it). And query understanding was recorded as an unbuilt dependency: the full structured key appears in question text 0% of the time (plan 31%, year 7%, benefit line 17%); in deployment the employer and plan join from the member record while the benefit line is inferred. F3 (Part I) plots the delivery story in these metric names.

The same stage closed three scope checks beyond the headline. The prose path was measured directly: retrieval given the resolved document reads 69%@1 / 86%@2 / 100%@10 on prose questions, with no regression from the re-chunk. Access control holds at the retrieval layer by construction: 0 of 180 deliveries crossed an employer boundary, because the structured key resolves within the asker’s scope before any ranking happens. And supersession exposure was sized: 37% of the table questions’ source documents carry amendments, and the amendments are entirely prose, so a minimal precedence scaffold shipped at this stage at zero test-set cost, with substantive amendment handling scheduled for the phase that could validate it.

5.2 Model capacity — re-asked after delivery was fixed (IV-3)

Phase 1 had measured capacity as a null. With delivery fixed, the question was re-opened from scratch, and the answer reversed. On the 82-question table-heavy evaluation, accuracy climbed cleanly with size (8B 86.6%, 14B 91.5%, 27B 92.7%, 35B 95.1%) and safety climbed with it: confident-wrong answers fell 11 → 7 → 4 → 4. A second capability threshold appeared in the selector role (the model-as-judge that would gate auto-accept if a model played that part): the two smaller models took 3 and 2 critical errors as cell selectors, the two larger took zero. That threshold is immaterial to the generator choice, since the deterministic spine is the gate (§5.3), but it is decisive for anyone tempted to let a mid-size model approve its own answers.

The constraint was the single GPU card, which the retrieval/scoring stack already occupies 6.7 GB of. The 35B fills the card by itself; expert offload fits it co-resident only at roughly 9 s/query; and the one 27B-class configuration that genuinely co-resided required an aggressive 3-bit quantization. That 3-bit 27B looked best on the 82-question sample; the full-scale 180-question robustness re-test overturned it, a reversal flagged against its own earlier pick. The 14B tied it on overall accuracy (93.3% each) and beat it by 4.7 points on the hard 43-question tabular slice (88.4% vs 83.7%), while running faster (1.49 s vs 1.97 s) in a plain 4-bit, full-context configuration with no exotic flags. Aggressive quantization bites exactly the numeric cells this engagement existed to fix, and it bit the sparse mixture-of-experts ceiling model even harder than the dense one.

That head-to-head yielded a reusable engineering rule, recorded for future hardware decisions: the 35B mixture-of-experts carries the highest full-precision ceiling in the ladder, yet at the 3-bit quantization that single-card co-residence demands it fell roughly five points, landing below the dense 27B, which lost nothing at all on the same evaluation. A sparse model activates only a few billion parameters per token, leaving fewer effective weights to absorb quantization noise; a higher ceiling does not survive the quant required to fit it.

Selection: the 14B dense model, 4-bit quantization, 16K context, co-resident with the full retrieval stack at 21.9 of 24.6 GB, ~1.5 s/query. The documented menu kept the options honest: the 8B as the conservative floor with the most headroom, and the 35B on a second consumer GPU as the optional quality ceiling, neither required for any committed number. The recorded lesson: small samples flatter aggressive configurations; validate the candidate at full scale, on the hard slice, before committing.

5.3 Typed extraction — the commitment layer (IV-4)

The work item said adapter retraining. The evidence said the adapter was inert on accuracy (Cohen’s d −0.021) and that its one contribution, citation-first formatting, was replicable in deterministic code at 100%. So the work shipped as typed extraction instead, documented as a change order recording the rationale.

The spine’s mechanics, end to end in code: parse the delivered tables into grids of row labels × column headers; match the question to a row (lexical + embedding, carrying a top-1-vs-top-2 selection margin) and a column (distinctive-qualifier, single-value-column, or value-agreement); read grid[row][column] verbatim; accept only if the answer is a single cell ∧ the row resolved ∧ the column resolved ∧ the value type matches the question’s stated form ∧ the value is concrete; otherwise abstain and route to a person with the source attached. A deterministic renderer then produces the citation-first answer and compliance footer — the adapter’s old job, now auditable, at 100%.

The ingestion surfaced exactly one critical error during development: a coinsurance-percentage question whose document also carried a flat-copay row for the same service, where treating money and percent as interchangeable let the wrong-form cell through. A strict-form type gate (when the question names the form, require an exact type match) closed it, restoring zero. The 14B arm recovers a few abstained cells but took one critical error of its own on a question the spine had right — the selector-safety threshold of §5.2 reproduced on the production base, and the reason the deterministic spine, not model judgment, is the committable gate (the model arm is coverage extension only, and was not deployed as the gate). On the single-cell numeric subset, extraction read 83% correct against generation’s 72%.

Integrated across the full set at this stage (still oracle-routed; production routing is §5.6), the system measured 82.2% → 95.0% answerable at the peak build (171/180), with benefit lookup at 76.2% → 94.3%. The net is +23: 27 baseline misses fixed, 4 regressions (one member-identity roster lookup the table path does not serve plus three computed/rule prose questions), each characterized individually. F7 (Part I) plots the census comparison.

Two more results round out the stage. All 13 auto-accepted answers sat in the top confidence bucket of the margin-derived calibration at 100% accuracy: the per-category calibrated-confidence mechanism demonstrated, with corpus-scale calibration scheduled for where the data to support it exists (§5.4). And the deterministic renderer measured 100% citation-first formatting with the required compliance footer on the accepted set, against the adapter’s measured 80–92%, verifying the change order’s claim.

5.4 The confidence rebuild — an honest null (IV-5)

The rebuild validated its feature thesis and then reported itself unnecessary. The structural gap was real: answer-text-only features rank the post-improvement residual at or below chance (leave-one-out AUC 0.30), while production-observable retrieval-quality and provenance features lift it to 0.58, a paired ablation gain of +0.28 with a 95% CI of [0.02, 0.58] excluding zero, with coefficients that read sensibly (citation adherence +0.86 and retrieval score +0.55 toward correct; value-similar family size −0.47 and amendment-in-play −0.35 toward error).

But the spine had starved the ranker. After §5.3 the system is 95.0% accurate at the peak build, leaving nine confident errors in 180; on nine negatives, absolute discrimination is statistically indistinguishable from chance (LOO-AUC 0.58, 95% CI [0.35, 0.79], containing 0.5) and the model does not beat a constant base-rate predictor on calibration (Brier 0.053 vs 0.048). Per-category calibration is negative-starved: only benefit lookup carries enough errors (7) to calibrate at all, the learned ranking there is at chance (0.49), and its zero-critical-error frontier computes to 1.6%, negligible against the spine’s 30%. The conclusion was reported as a finding: the committable auto-accept gate is the deterministic spine, unambiguously; the learned model’s durable value is the validated feature framework and the per-category calibration mechanism, positioned to earn discrimination at corpus scale where negatives densify. The confidence model got harder precisely because the system got more accurate.

5.5 Ingestion fidelity — the vision-model path (IV-6)

The spine guarantees correct selection, not cell fidelity: if the ingested text is already corrupted, the spine reads the corruption verbatim. This is the one place zero-critical-error is not protected by construction, and the binding dependency for auto-accept coverage on a document library roughly 30% scanned and about 10% degraded. Measured on the 11-question single-cell census: character-level OCR at light degradation produced 2 of 11 confidently wrong reads (a 75-dollar copay transcribed as “7S”, a 50-dollar copay transcribed as “l0% after deductible”), and an independent re-draw on a different seed produced four confidently wrong reads across its light and heavy conditions, so the failure is seed-robust, not an unlucky draw. Structure-destroying degradation is comparatively benign (the grid fails to parse, the spine abstains, coverage drops safely); character-level noise is the silent killer, because a corrupted value still looks type-valid.

The recovery path reads the image instead of the extracted text: render each delivered grid to a realistic table image; degrade it across an 11-condition scan-robustness sweep (a clean control; realistic photocopy chains at three severities; isolated blur, noise, JPEG, and low-resolution stressors; skew, ink bleed-through, and harsh binarization); have an 8B-parameter vision-language model transcribe it back under a verbatim-or-“[illegible]” instruction (never guess, never autocorrect); then run the identical spine. The result: zero confidently wrong reads across all ten degraded conditions, with each cell character-OCR corrupts read correctly in 10 of 10 degraded conditions, and the single imperfect case a safe abstain (bleed-through rendered one cell as a placeholder, which the spine’s concreteness gate refused). The vision model fits the same single card (bf16, ~17.5 GB, ~6 s/table), so ingestion needs no second GPU either.

The boundary was stated at the time: this is synthetic degradation of rendered clean replica tables, a validated mechanism rather than a corpus coverage claim, and it did not move the held auto-accept prediction. Phase 3 is where the mechanism met the production library, recovering the 162 scanned documents behind the corpus validating run (§7).

5.6 The router — removing the last oracle (IV-7)

Every integrated number through §5.3 routed questions to the spine or the generator using an oracle that inspected the gold answer’s source passage — legitimate for isolating components, impossible in production. IV-7 therefore rebuilt routing from production signals only: resolve the structured key, score the question against the resolved document’s table descriptors, and route to the spine iff the top descriptor cosine clears a threshold chosen on the router’s own precision/recall curve, never on system accuracy.

The router itself is mediocre, and the report said so: F1 0.59 at the optimal threshold (precision 0.47, recall 0.79), below the 0.72 the same design had scored on the old index, ironically because the ingestion fix produces clean, isolated descriptors that make benefit questions answered in prose look like benefit-table questions. And it does not matter:

The error decomposition is the finding. Of 38 false-positive routes (prose questions sent to the spine), the spine abstained on 37, and the one it accepted was correct: zero critical errors, zero accuracy lost. Of 9 false negatives (tabular questions sent to generation), the generator still answered 8 of 9 correctly: zero accuracy lost, two auto-accepts forgone (cosines 0.518 and 0.529, just under the threshold; both recovered at a lower threshold at no critical-error cost). Auto-accept under the router: 6.7% of answerable (12/180) at 100% precision and 0 critical errors, versus 7.2% (13/180) under the oracle and 30% of the 43-question tabular census — one zero-critical-error property, three denominators, never blended. The headline is threshold-robust (the peak 95.0% and 0 critical errors at every τ from 0.50 to 0.65), and 0 of 20 adversarial questions routed to the spine at any setting. Conclusion: the deterministic spine, not the router, is the safety gate; the router is triage, to be tuned for recall in production where the spine makes false positives free.

Two boundary notes. The descriptor-cosine signal is real but overlapping (tabular questions median 0.614 against prose 0.354), and the false positives are not noise: every one carries a benefit line, and 21 of the 38 are benefit-lookup questions whose answers happen to live in prose. No gold-free signal short of reading the answer separates “benefit, answered in a table” from “benefit, answered in prose,” which is why router precision is structurally capped and the spine has to be the gate. And the all-200 secondary number (79.0% → 90.5%) conservatively inherits the baseline’s behavior on the 20 unanswerable questions, because this recombination did not re-measure the generative refusal path; under the all-200 accuracy label that behavior credits 10 of the 20, a stricter standard than the refusal-detection instrument, whose figures (90.0% baseline, 95.0% improved) are measured and reported separately (§2.4).

5.7 Attribution — where the lift comes from (IV-8)

The entire accuracy lift is retrieval plus model capacity; the spine and the router add zero points. That is the architecture’s division of labor: the spine auto-accepts exactly the clean cases the capable reader already gets right, converting accuracy that exists into answers that can be committed (deterministic, citation-first, zero-critical-error) — a different property, earned by a different component, and reported separately throughout this engagement. The attribution also explains the program’s two nulls: the confidence model sat at chance (§5.4) and the router’s F1 was harmless (§5.6) because the accuracy was banked upstream, where it is least fragile. One method note: the split within the retrieval+capacity bundle is cross-referenced from §5.2’s isolated measurements rather than re-run end-to-end, and is recorded as such.

The lift was broad: every category, difficulty band, and tier improved or held — benefit lookup +18.1 pp (76.2% → 94.3%), the weakest baseline band (difficulty 2) +17.0 pp (73.2% → 90.2%), the deliberately hard edge-case tier +12.5 pp (85.0% → 97.5%) — and no slice regressed. The blended results carry two flags taken up next: automated-NLI faithfulness reads 93.3% on the peak build, below the ≥95% committed bar (the diagnosis, the lever sweep, and the recorded election are §6), and hallucination reads 1.7% against a 0.0% baseline, a regression confined to the generative path, with the auto-accept path at 0.0% by construction. F5 and F6 (Part I) plot the category lift and the attribution waterfall.

6. The Faithfulness Trade-off and the Election

One committed prediction tracked below its bar on the automated scorer, and what followed (diagnose the miss, attack the diagnosis, sweep every alternative, route the decision before the gate, record the recommendation and the election separately) is the engagement’s governance model compressed into a single metric. Part I tells this story in narrative form (“The Metric Trade-off and Configuration Selection”) and F8, embedded there, plots the frontier; this section is the technical record behind both.

6.1 The 93.3% reading and its diagnosis

Citation faithfulness was a committed prediction at ≥95% on the automated NLI scorer, a maintain bar registered against the baseline’s 96.1%. The peak build read 93.3% blended. The gap sits entirely on the generative path: the auto-accept path is 100% faithful by construction (it reads the cell verbatim and renders citation-first), while the 168 generated answers scored 92.9%, twelve answers flagged unfaithful.

The twelve decompose into two clusters. About nine are correct, properly grounded benefit values restated in prose: their answer-to-passage lexical overlap is 0.08–0.15 (a clean sentence scored against a markdown table), and their NLI adherence lands at 0.49–0.70, just under the 0.70 faithfulness threshold. About three are genuine: one outright wrong answer, the others conflicts. So roughly three-quarters of the measured “faithfulness loss” is the scorer failing to align a correct table-to-prose paraphrase, not the system asserting something its source does not support.

“The scorer is wrong” is what every vendor says about a missed metric, so the diagnosis was pressure-tested against its obvious rival before anyone was asked to accept it. The natural suspect was the compliance footer: boilerplate appended to 99% of answers, plausibly read by an NLI model as unsupported claims. The hypothesis was tested directly and refuted: stripping the footer before scoring made automated faithfulness worse, 93.3% → 91.7%, and pushed measured hallucination from 1.7% to 3.9%; the scorer already handles the footer, and removing it leaves only the substantive borderline claims. (That 91.7% is a faithfulness reading on the assessment corpus; it is numerically coincidental with, and must never be conflated with, the 91.7% raw corpus accuracy of §7.1.) The artifact diagnosis survived because the refutation was attempted first.

6.2 The lever, and the sweep that proved the trade irreducible

A viable lever existed: instruct the model to quote the grounding value verbatim before explaining, the generative analog of what the deterministic renderer already does on the auto-accept path. Applied through the production router, it lifts automated faithfulness over the bar. It is not free, and the deployable-policy sweep below is the evidence the decision was made on:

The sweep’s recorded verdict: no deployable policy reaches the peak build’s accuracy and the ≥95% automated-faithfulness bar simultaneously. The lever fixes four flagged questions and breaks exactly one: Q-012, a health-savings-account contribution question whose correct answer is a computed total. Verbatim quoting makes the model quote the two source cells faithfully and drop the sum — the one number the member actually needs — turning a correct answer into an incomplete one. And those five questions are indistinguishable on every signal the live system can see: the same computed-multipart question class, the same spine confidence, overlapping router cosines. No deployable rule keeps the four fixes without taking the one break, and the oracle row needs the gold “this was a table question” label that production does not have. Option B therefore trades one, member-facing regression (caught by the verification mandate) to clear what §6.1 shows is substantially a scorer artifact.

6.3 Routed before the gate; recommended; elected

Before the second gate, the entire picture was considered: the reading, the diagnosis, the refuted footer hypothesis, both validated configurations with their exact costs, the irreducibility sweep, and a recommendation. The routing mattered as much as the content: a committed metric tracking below target reached an analysis with options while there was still time to think, not as a slide in the gate room (§2.2).

At the gate, Option B was selected, the balanced build at 94.4% accuracy / 95.6% automated faithfulness, on defensibility grounds: a configuration that meets every committed bar on the scorer that runs on every build is defensible to an auditor in one sentence, and the six-tenths of a point it costs is caught by the verification workflow that runs anyway. Both positions, the recommendation and the election, were recorded in the gate decision with their reasons.

Three attachments made the elected configuration safer than it had been an hour earlier: the regressed computed-total question went onto the Phase-3 work list as a refinement (condition C-2); a one-time human re-rating was retained as corroboration, filed alongside the automated number, not as the pass dependency (condition C-3); and the recommendation-versus-election record itself (condition C-1).

6.4 The recovery

The refinement then closed the gap from the other side. Midway through Phase 3, C-2 taught the extraction path to distinguish a single-cell read from a computed total, recovering Q-012 without reopening the faithfulness exposure. The production build moved to 94.8% accuracy / 95.5% automated faithfulness, the deployed configuration, measured at acceptance on the expanded 300-question production instrument (§8.2) — effectively the peak build’s accuracy with the committed faithfulness bar clearing outright on the scorer of record. The human re-rating ran on the sample aligned with the automated pass. On F8’s frontier (Part I), the deployed point sits where the gate-week sweep said no deployable configuration could yet reach, moved there by engineering the one question that forced the trade (and read on the acceptance instrument rather than the pilot one).

7. Productionization and Operations (Phase 3)

Phase 3’s job was to make the measured result a system that could run: transfer the numbers to the corpus, and build the production path and its compliance controls. It ran against the conditions register of §2.2, with the only incident landing in the middle of it, handled by logic built for exactly that event.

7.1 The corpus validating run (A-1)

The first Phase-3 work — ahead of any production build — was the validating run the second gate had made a condition: re-run the deployed before/after against the full production document library and re-score all nine predictions, on thresholds defined before the run. This was the engagement’s #1 caveat (§2.5) being retired on the data that mattered.

The raw read was reported as exactly that, with its cause: 162 scanned plan documents, about 18% of the plan-document library, had been ingested below the character-OCR legibility threshold. This was the failure class the assessment had stress-tested (§4.4), the Week-1 fidelity check had held auto-accept for (§2.3), and IV-6 had built the recovery for on synthetic degradation with the explicit boundary that coverage would be measured here (§5.5). The vision-model path re-read the 162 documents’ tables from images, and the validating run landed within noise of the replica result, with the zero-critical-error auto-accept property surviving on data (~30% of the tabular slice, 0 critical errors). Both numbers, raw and recovered, went in the status report with the mechanism in between.

The evaluation instrument also grew here: ~100 additional questions, stratified to the same category mix, de-duplicated against the original set by embedding similarity, gold answers agreed, inter-annotator agreement κ ≈ 0.79 on a sample. The expanded 300-question set re-confirmed the result and became the acceptance instrument for the third gate (§8.2).

7.2 The production build, and the one incident

The production stand-up followed: cutover early in the phase (production document-store and wiki wiring, finalized ACLs, live ingest webhooks), and a CI/CD pipeline whose stages are the engagement’s controls expressed as code (lint → unit tests → the ACL compliance harness → a PII-detection regression → integration → staging → acceptance → a 10% canary with automated rollback), with every promotion requiring a promoter, and rollback a single feature-flag change to the prior approved artifact. The C-2 refinement (§6.4) and the router recall-tuning (condition A-4) were promoted through exactly this path.

The incident: midway through the phase, under sustained load, the canary node (the 10% of traffic running the newest build) exhausted GPU memory and stopped completing requests. The automated rollback tripped as designed and returned that traffic to the stable build inside three minutes; the request logs confirmed zero visible impact. The incident was classified at the highest severity on detection, the worst plausible read of a GPU out-of-memory, then reclassified one step down, with the reasons stated: rollback confirmed, affect would be on the canary node only, no compliance, data-loss, or availability dimension. Closed with the fix promoted through the same canary it had tripped.

The root cause was the engagement’s own Phase 2 finding meeting production concurrency: KV-cache growth under concurrent traffic pushed resident VRAM past the single card’s envelope, the headroom limit IV-3 had characterized in the lab (§5.2). The fix was a concurrency cap plus a KV-cache budget keeping resident VRAM under the card’s dedicated envelope; the durable part was a new VRAM-headroom alert wired into monitoring, turning a characterized limit into an enforced one. When the sponsor asked the question the incident begged (is the single card undersized?), the answer was no: with the cap in place the one card carries the committed load with headroom, and the second-card option stayed exactly where the gate had left it, a backlog item triggered only if the deferred endurance measurement shows a shortfall (§8.2). The failure mode did not recur for the remainder of the engagement.

7.3 The compliance surface

The P-1 answer-scrub — the hard gate. The answer-level PII scrub and input-redaction controls had been converted at the second gate from “demonstrated at DG2” into a hard pre-go-live condition, written into both the condition register and the SOW’s go-live criteria: no member answer reaches production until Compliance accepts the controls. They were built into the production answer path during Phase 3, measured at 99.7% detection recall against a ≥99.5% bar on a fresh ~200-query holdout (the single miss: a partial identifier embedded inside a longer plan code, the residual pattern already characterized and accepted), and accepted in the gate run-up. The date stood because the control was ready. Per-category recall on the free-text identifier classes was delivered alongside, with confidence intervals and a secondary detector on 100% of log writes (condition P-2); the per-category floor refused at the first gate stayed refused, with the measured-residual-with-CI structure serving in its place.

Access control, at three distinct scopes, kept distinct because they are different instruments:

The ACL-failure alert is the monitoring stack’s zero-tolerance page.

The audit posture. The SOC 2 control-mapping evidence pack, accepted in the gate run-up, maps the system into the SOC 2 Type II control catalog instead of standing up a parallel framework, across security, availability, processing integrity, confidentiality, and privacy; a cross-reference index ties each control to its corresponding privacy-regulation citation so one set of evidence serves both assessors. Underneath it: a searchable audit store carrying every query, retrieval, answer, refusal, and feedback event at seven-year retention with a PII pre-scrub on the searchable record; the un-redacted log encrypted with dual-control decryption; and random-sample audits of the searchable log returning zero protected or personally identifying tokens. The corroborating human re-rating of §6.4 is filed in the same pack, alongside the automated number.

7.4 Operations hardening

The disaster-recovery drill ran late in the phase against the warm-standby host, initiated from the worst realistic state: a staging run was in progress on the standby (its designed dual-use posture), so the cutover began with the teardown step a clean-lab drill would skip.

The one-hour recovery-point objective was met via the hourly-incremental backup and buffered-write replay. The drill’s product was its deviations, both small and both fixed: the standby’s pre-pulled images were two release tags behind (the pre-pull cadence is now kept within one tag of production), and one manual endpoint-update step was missing from the runbook page (added). The largest single contributor, the ~55-minute image sync and model load, is explained in the drill report and has an identified lever, which is what makes the next cutover faster than this one.

Monitoring ships as 15 production metrics: seven per-query (per-stage latency including the route and typed-extraction stages, retrieved-document provenance, the model/config version that served the answer, which path the answer took and its selection margin, outcome, hashed identity, feedback) and eight daily aggregations (volume, rating distribution, the fallback-to-manual-search rate as the adoption signal, a rolling 7-day flagged-incorrect rate as the quality signal, cache hits, queue waits, load failures, PII-pipeline errors). 9 alert rules sit on top: ACL failure and PII-pipeline error as zero-tolerance critical pages, model/config load failure blocking any promotion, quality-degradation, volume-anomaly, latency, queue-wait, and cache-health rules, and the ninth: the VRAM-headroom early warning the canary incident created.

7.5 The deployed topology

Eight containers on the single existing GPU, co-resident at a measured ~93% VRAM capacity; the remaining headroom is the budget the ninth alert protects.

The vision-model table reader (~17.5 GB) cannot co-reside with the serving stack and does not need to: it runs as an on-demand ingestion-time model during reindex and reconciliation windows, time-sharing the GPU; it is not a ninth persistent container, and not a second card. Appendix B carries the full deployed-stack reference card.

8. Decision-Gate Outcomes and Pre-Registered Predictions

8.1 Three gates, three outcomes

By the third gate, every prior condition was either discharged in evidence or carried forward explicitly: the redaction control accepted (§7.3), the C-2 refinement deployed (§6.4), the corroborating re-rating filed.

8.2 The third gate in detail

The twenty acceptance criteria were fixed in advance, at the production threshold, and scored in the open; the documentation carries all twenty with their measurements. The acceptance window itself ran as a shadow-mode deployment (2,813 questions answered in parallel with the existing workflow, reversible at any moment), the same pattern that governs every future model change. The bars:

Two measurement notes traveled with the acceptance record. The shadow window’s traffic ran at roughly 563 questions per day, below the volume the discovery-period baseline projected, because the window coincided with a reduced call volume; the acceptance measurements are not volume-gated, but the note was filed so the first volume reading would be interpreted against the discovery baseline, not the acceptance-window rate. And where the criteria are tighter than the registered predictions, both bars were stated: hallucination, for example, was accepted at 1.7% against a ≤2% bar, inside the looser ≤3% bar the prediction had registered.

The deferred criterion — criterion 18, six weeks of sustained endurance at three times baseline load — was neither waved through nor failed. Phase 3’s synthetic-load budget could not honestly simulate six weeks of peak volume, so the criterion was deferred to the peak window itself: recorded in the gate decision, scheduled, and assigned a reporting date (the season-end review).

The gate run-up also produced the engagement’s most instructive small governance moment. RCTK found, in its own success-criteria baseline, a stale faithfulness bar of ≥97%: a clerical inheritance from the earlier build-from-scratch scoping that contradicted the ≥95% maintain bar actually registered in writing at kickoff and scored at the second gate. The correction was made toward the registered commitment and footnoted in the gate pre-read.

The third gate’s one decision was the auto-accept graduation, and it went with a deliberately narrow recommendation: graduate the single-cell benefit-lookup category, where the verification record showed zero critical errors, and keep everything else in assist mode.

8.3 The nine predictions, scored

Nine predictions were registered at the Phase-2 kickoff, tiered committed/stretch, to be scored at the gates pass-or-fail regardless of outcome (§2.3). The mid-engagement reading was reported in the same spirit: at the second gate the scorecard went out as five tracking-pass, one at-risk (faithfulness, §6), one held (auto-accept, then at its fourth hold), and two deferred-by-design (the protected-information control, built before go-live by condition; time savings, measurable only on calls) — not as “all on track.”

Nine of nine finished on the right side of their registered bars.

9. Production Results — The First Support Cycle

The engagement’s numbers were measured one more time. The first support period ran under load ramping, measured by telemetry plus a 150-query audited gold-label sample. The table below sets the period against the committed bars.

The period was also the deferred criterion-18 endurance window (§8.2): sustained throughput tracked comfortably above the 3× target (the single card runs at a small fraction of its measured ceiling even at the volume ramp), and the formal call still waits for the complete window.

The period exercised the machinery for the one constant in a benefits corpus: change. When three next-plan-year documents finalized, the twelve updated plan documents were re-ingested under the table-aware pipeline, the structured-key index was updated, and, the step that matters, the precedence layer was set to serve each new plan year and suppress its superseded predecessor through the renewal-overlap weeks when both legitimately coexist. The refresh was validated against the held-out evaluation sets retained for exactly this purpose: correct resolution on the updated documents, zero stale-version serves on the spot-check, no regression on the broader slice, and the auto-accept property preserved on the refreshed cells — then promoted via the warm standby with zero downtime.

And the support period summary carried one service item: a false-abstain pattern, the system declining a narrow class of multi-year deductible questions during the renewal overlap that a tightened precedence signal would let it answer safely. The failure is conservative, costing coverage, not safety; it was logged, sized, and routed to the improvement backlog and the period-end review. The support draw itself was light (roughly a third of the budgeted hours), the expected shape of a clean period: the support tier is sized for the volume peak.

10. Evidence Index

Per the anonymization terms of this study report, artifacts are cited by descriptive title only. Every quantitative claim in this report traces to one of the artifacts below; the figure-data file published alongside this report carries every plotted value with a pointer to its source.

A published copy of this study report carries this index in descriptive form only; the underlying artifacts remain internal to the engagement record.

Appendix A — The Pre-Registered Prediction Protocol

The protocol is the engagement’s accountability instrument, and it is simple enough to reuse:

• Register before data. The predictions are written down at kickoff, before any pilot data exists, with the exact thresholds the gates will score.
• Tier by evidence, not by optimism. Committed means the diagnostic evidence is in hand and a miss is a serious, reportable event carrying a root-cause analysis, with no after-the-fact reclassification. Stretch means a specific uncertainty remains and a miss is informational.
• Commit thresholds with margin, never points. Accuracy was committed at ≥85% with an expected landing around 90–91%, so ordinary measurement noise could not put the commitment itself at risk.
• Name each held prediction’s dependency. Auto-accept was held on scanned-table ingestion fidelity; time savings on the measured search-versus-read split. A hold with a stated dependency reads as engineering honesty; a vague hold reads as hedging.
• Promote only when the evidence arrives. One promotion per arrival, attributable to the evidence.
• Score everything at the gates, pass or fail, regardless of outcome. The scorecard is reported in whatever state it is in (§8.3’s mid-engagement reading: five tracking-pass, one at-risk, one held, two deferred-by-design).

The full promotion ledger is short: numbers moved rarely, and only on arrivals.

The auto-accept prediction is the protocol’s worked example, because it was the one the room kept inviting RCTK to promote: five invitations, five holds.

The resolution: graduated at the third gate for exactly one category on the four-week zero-critical-error record; then ~1,100 live auto-accepts at zero critical errors in the first month (§9). The prediction finished as the protocol intends: the property committed early and kept, the volume earned late and granted narrowly.

The effect is gates without surprises: every number a gate scores was set earlier; every promotion happened between gates, on evidence, in writing; the one metric that went at-risk reached an analysis with options before the gate met. And each hold along the way is what made the committed numbers believable when they arrived.

Appendix B — Deployed Stack Reference Card

The configuration as accepted at the third gate and operated through the first support cycle.