Reclaiming AI Document Search Quality Through Configuration Testing And Parameter Sweeps

1. Executive Summary

Four iterations of RAG benchmark optimization (v5–v8) were conducted on the ExpertQA dataset using a 6-step retrieval-augmented generation pipeline powered by Nemotron-Nano-9B-v2-NVFP4 via vLLM. The study processed 14,545 benchmark requests plus 1,598 LLM judge evaluations over 5 days, progressively narrowing the configuration search space through a decision funnel: technique selection (v5), parameter optimization (v6), measurement reform (v7), and helpfulness assessment (v8). Each version answered a specific question, and its findings constrained the next iteration’s search space. The final recommendation is maxtok_768, a sentence_window_1 configuration with max_tokens=768, temperature=0.1, top_k=5, reranking disabled, and a detailed prompt template—which achieves the best balance of automated metric quality (composite rank #2, score 0.5823) and human-perceived helpfulness (answer relevance rank #2, score 4.365/5.0).

Final recommendation: The maxtok_768 configuration is the recommended operating point across both automated composite metrics and LLM-judged answer relevance. See Section 8 for the complete configuration card.

2. Methodology

2.1 Pipeline Architecture

All four versions share a common 6-step RAG pipeline:

1. Validate Input — Schema checks on question + ground truth
2. Retrieve — Dense retrieval via ChromaDB (Qwen3-Embedding-0.6B, 1024-dim) with optional BM25 hybrid fusion
3. Rerank — Cross-encoder re-ranking (bge-reranker-v2-m3) with configurable enable/disable
4. Generate — LLM answer generation via Nemotron-Nano-9B-v2-NVFP4 (vLLM, Marlin backend, RTX 4090)
5. Evaluate — Automated metrics computed against ground-truth answers and retrieved contexts
6. Compare — Pairwise statistical analysis across configurations

Dataset: ExpertQA: 150 stratified test cases drawn from a multi-domain expert-sourced QA corpus (medicine, law, science, technology, etc.). The same 150 test cases were reused across all versions for comparability.

Infrastructure: AI Manager service orchestrating Docker containers. Single Nemotron-9B NVFP4 instance (~12.2 GiB model, ~23 GiB peak VRAM). PostgreSQL + Redis backing stores. All runs on a single RTX 4090 (24 GiB).

Embedding model: Qwen3-Embedding-0.6B with asymmetric query/document encoding. Chunked documents stored in ChromaDB collections with per-technique indexing.

2.2 Evaluation Metrics

NLI Faithfulness was dropped after v6 due to an 88.8% ceiling effect and independent evaluation model concerns. Answer Relevance was added in v8 as a holistic helpfulness measure orthogonal to component metrics.

2.3 Composite Score Design

The composite score evolved across versions as metric quality findings accumulated:

Key changes:

• v7 re-evaluation dropped NLI (ceiling) and redistributed its 0.20 weight to utilization (+0.05) and completeness (+0.10)
• The v7 re-evaluation composite (template_v2_no_nli) was selected for strongest discrimination (F=6.81 vs F=3.80 for original weights)

2.4 Statistical Framework

All versions use a consistent statistical framework:

• Paired tests: Wilcoxon signed-rank (non-parametric) for pairwise comparisons on the same test cases
• Multiple comparison correction: Holm-Bonferroni step-down procedure
• Effect sizes: Cohen’s d with benchmarks: |d| < 0.2 = Negligible, 0.2–0.5 = Small, 0.5–0.8 = Medium, > 0.8 = Large
• Omnibus tests: Friedman test (non-parametric repeated measures) for rank-based group differences; one-way ANOVA as parametric complement
• Confidence intervals: 10,000-iteration bootstrap (BCa) for technique means
• Bayesian evidence: BIC-based Wagenmakers (2007) Bayes factors
• Construct validity: Pre-registered directional predictions tested as strict (must hold) or informational (expected but not required)
• Discrimination power: ANOVA F-statistics and eta-squared (η²) for between-technique variance
• Ceiling/floor analysis: Percentage of observations at metric bounds; >50% classified as “Unacceptable”

Note on effect sizes: Cohen’s d is computed on raw paired score differences (not rank-transformed), which is standard practice when the underlying continuous metrics have interval-scale properties. While rank-biserial correlation would be the natural nonparametric companion for Wilcoxon signed-rank tests, Cohen’s d was chosen for interpretability and comparability with power analysis conventions. Kendall’s W was not reported alongside Friedman tests but would be a useful addition in future iterations.

3. v5 — Technique Selection

Question: Which chunking technique produces the best RAG responses on ExpertQA?

Design: 4 techniques (semantic, sentence_400, recursive_400, sentence_window_3) evaluated at default parameters with 150 samples each (600 main comparisons), plus 8 parameter sweeps (chunk_size, prompt_template, final_top_k, similarity_threshold, reranking, retrieval_method, adherence_support, chunk_size for sentence_window). 7,350 completed requests, 99.0% success rate. Runtime ~6.3 hours.

The results revealed a fundamental fidelity-coverage tradeoff: semantic excelled at adherence (0.733) and NLI faithfulness (0.994), while sentence_window_3 led on relevance (0.808) and utilization (0.836). The gap was substantial, with Medium effect sizes (d=0.64–0.73) on coverage metrics.

Construct Validity (v5): 4/5 Strict PASS

Test 3 failed because reranking hurts on ExpertQA, unlike techqa/covidqa. This was an important dataset-specific finding carried forward.

Metric Quality Alerts

• NLI Faithfulness: 88.8% ceiling — unreliable as discriminator (recommended drop in v6+)
• Completeness: ANOVA F=1.77, p=0.15 — zero discriminative power; eta-squared=0.009 (negligible)
• Relevance: 37.3% ceiling (sentence_window_3 at 67.3%) — marginal

Decisions Carried Forward

1. Technique = sentence_window— selected over semantic (adherence leader, 0.733) because sentence_window provides the best coverage-fidelity balance, is fastest (13.1s vs 14.8s mean), has the lowest failure rate (0.3% vs 0.8%), and its window_size parameter enables further optimization in v6. All 4 techniques exceeded the 0.60 adherence threshold and were retained as viable candidates; the v6 carry-forward decision was which to prioritize.
2. Investigate window_size=1 — highest adherence (0.753) and NLI (0.982) among all configs
3. Reranking = investigate further — counterintuitive negative effect needs confirmation
4. Flag completeness and NLI as non-discriminating — under binary formulation, neither metric discriminates between techniques; reassess after measurement reform
5. Dense retrieval only — hybrid severely degraded fidelity

4. v6 — Window Size and Parameter Optimization

Question: Which sentence_window size and parameter combination maximizes composite quality?

Design: 4 window sizes (1, 2, 3, 4) at 200 samples each, plus 3 parameter sweeps (final_top_k, reranking_enabled, prompt_template). 5,597 completed requests out of 5,617 submitted (99.6% completion rate; 5 failures, 15 other non-completed). Runtime ~9.2 hours across 4 runs (1 main + 3 sweeps).

Window_size=1 won the composite ranking through leading fidelity metrics (adherence, NLI). The fidelity-coverage tradeoff from v5 repeated at finer granularity: sw_1 led on adherence (+0.104 vs sw_4, d=0.42) while sw_3/sw_4 led on utilization (+0.054, not significant).

Disabling reranking improved composite by +0.041, confirming v5’s counterintuitive finding. The effect was driven primarily by relevance (0.784 → 0.912, +0.128) and utilization (0.818 → 0.849). A plausible mechanism is that the cross-encoder filtered out chunks that were topically relevant but not the closest match—on ExpertQA’s diverse multi-domain content, this filtering may have been counterproductive.

Sweep Details

Top_k sweep (k = 3, 5, 7, 10): Composite flat across k=5–10 (0.749–0.755). k=3 dropped to 0.722. Decision: k=5 retained (default, balanced).

Reranking sweep: Disabled significantly better (+0.041 composite). Bayesian evidence for the overall window-size adherence gradient: BF10=308.6 (extreme) for sw_1 vs sw_4.

Template sweep: Detailed template +0.024 composite over default. Chain-of-thought comparable but 2.1× slower (~29s vs ~14s mean generation time).

Construct Validity (v6): 6/6 PASS (Highest of Any Version)

Ceiling Problem Foreshadowing

Despite 6/6 construct validity (the highest of any version), v6 revealed a severe measurement problem:

Three of five metrics had unacceptable ceiling effects (>50% of observations at maximum). This meant the composite score was increasingly driven by the few metrics with remaining variance, raising the question: is this optimizing metrics or actual quality?

Decisions Carried Forward

1. Window size = 1 — best composite, fidelity leader
2. Reranking = disabled — confirmed negative effect
3. Template = detailed — modest gain, retained
4. top_k = 5 — flat region, no need to change
5. Address ceiling effects — measurement reform needed before trusting generation-level optimization

5. v7 — Measurement Reform

Question: Are binary metrics hiding meaningful differences between generation configurations?

Design: 8 named generation configurations varying prompt_template (detailed, grounded_detailed, precise, structured), temperature (0.0 vs 0.1 default), max_tokens (768 vs 512 default), and top_k (3, 5, 7). All used sentence_window_1 with system_prompt=/no_think and other defaults from v6. 1,598 completed requests (200 per config, minus 2 missing). The same responses were scored twice: once with original binary metrics, once with continuous reformulations using an independent evaluation model.

The 8 Configurations

All configs share: system_prompt=/no_think, reranking=false, dense retrieval, similarity_threshold=0.45.

The Plateau Discovery

With binary metrics, the top-4 configurations (topk_3, maxtok_768, temp_greedy, anchor_detailed) scored 0.830–0.832 composite — a spread of just 0.0017. The ranking appeared converged: further optimization seemed pointless.

But investigation revealed this convergence was an artifact of metric ceilings, not genuine equivalence.

The binary relevance metric had 86.7% of all observations at the ceiling (1.0). Any configuration that exceeded the threshold got a perfect score, erasing all differences above that line. The continuous formulation, using raw cosine similarity instead of threshold-based binary classification, restored the full measurement range.

The top-4 order was preserved, but the spread widened from 0.0017 to 0.0036, though still narrow. The continuous composite revealed 16 significant pairwise differences (vs fewer with binary), with 5 Large or Medium effects (4 with d > 0.8, plus structured vs topk_7 at d = 0.71), all involving structured as the worst config.

Construct Validity Revolution: 0/2 → 2/2

Under binary metrics, both strict construct validity tests failed — the threshold-based scoring masked the expected monotonic relationship between top_k and relevance. After continuous reformulation:

This was the strongest evidence that binary metrics were not just noisy but actively misleading — they could not detect real differences that continuous metrics revealed.

The Top-4 Cluster Problem

Even with continuous metrics, the top-4 configs (composite 0.579–0.583) remained statistically indistinguishable in pairwise tests (all Holm-corrected p > 0.05, all d < 0.15). The composite metric had reached its discriminative limit: the four best configurations produced responses of equivalent measured quality across adherence, relevance, utilization, and completeness.

This raised the question: is there a quality dimension not being measured?

Decisions Carried Forward

1. Continuous metrics adopted — binary formulations permanently retired
2. NLI dropped — 88.8% ceiling, independent eval model concerns
3. Top-4 cluster identified — topk_3, maxtok_768, temp_greedy, anchor_detailed
4. Need a new discriminator — automated composite cannot break the tie
5. Explore answer relevance — holistic helpfulness judgment as complementary signal

6. v8 — Answer Relevance Judge

Question: Can an LLM-judged answer relevance score discriminate between the top-4 configs that automated metrics cannot?

Design: Qwen3.5-9B (BF16 via vLLM, Docker container) scored all 1,598 v7 responses on a 1–5 Likert scale for answer relevance — “how helpful and relevant is this answer to the user’s question, considering completeness, accuracy, and specificity?” Temperature=0.0 for deterministic scoring. Scoring time: 969 seconds (1.5 responses/second).

Judge Calibration

Before scoring, the judge was calibrated against 50 human-labeled samples:

Kappa fell below the 0.6 threshold, indicating moderate (not substantial) agreement. However, three factors supported proceeding: (1) 88% of judgments were within 1 point of human labels, (2) Spearman correlation was significant (p<0.001), and (3) the primary analysis uses relative ranking rather than absolute scores, which is robust to systematic bias.

The AR-Composite Tension

The most striking finding was structured: ranked dead last on composite (#8, score 0.552) but first on answer relevance (#1, AR 4.44). This suggests that the structured template produces responses that users find helpful despite scoring poorly on embedding-based metrics. This tension is informative but does not change the recommendation; structured’s composite deficit (d > 0.9 vs top-4) is too large to justify on automated metrics alone.

Top-4 Discrimination: AR Breaks the Tie

The central question: does AR discriminate within the top-4 cluster that composite cannot separate?

Overall Friedman test (all 8 configs): chi2 = 47.12, p = 5.3e-08. AR discriminates across the full set.

Top-4 Friedman test: chi2 = 12.41, p = 0.006. AR successfully discriminates within the top-4 cluster.

maxtok_768 significantly outperforms both anchor_detailed (d=0.221, p=0.014) and topk_3 (d=0.218, p=0.013). These are Small effects, but they are the only statistically significant differences within a cluster that composite could not crack.

AR is Not a Proxy for Existing Metrics

Correlations between AR and automated metrics were modest:

The strongest correlation (rho=0.331 with composite) confirms AR captures signal not present in automated metrics. If AR were simply a noisy version of composite, one would expect rho > 0.7.

Construct Validity (v8): 1/2 Strict PASS

Test 2 failed by a negligible margin (+0.019, d=0.03). The grounded_detailed template performed slightly better than expected, possibly because its emphasis on sourcing improves perceived answer quality, though this was not directly tested.

Decisions Carried Forward

maxtok_768 is the final selection. It is the only configuration that ranks in the top-2 on both dimensions: composite rank #2 (0.582) and AR rank #2 (4.365). It significantly outperforms 2 of 3 other top-4 members on AR while maintaining equivalent composite performance.

7. Cross-Version Synthesis

Optimization Trajectory

The four versions trace a clear narrowing funnel:

v5: 4 techniques x 8 sweeps = 32 cells -> sentence_window selected
v6: 4 windows x 4 sweeps = 16 cells -> sw_1 + reranking=false
v7: 8 configs x 2 formulations = measurement reform -> top-4 cluster
v8: top-4 -> maxtok_768 (AR tiebreaker)

Each version reduced the search space by roughly 4× while adding measurement sophistication. The total optimization path evaluated 14,545 benchmark requests plus 1,598 LLM judge evaluations (16,143 total) across more than 40 unique configurations.

Metric Evolution

The evaluation framework evolved significantly through evidence-based decisions:

Construct Validity Trend

The v7 binary → continuous transition is the most dramatic result: the same data, same configurations, same predictions went from 0% to 100% validity solely through metric reformulation. This is strong evidence that metric design is at least as important as pipeline design.

Diminishing Returns

The composite gains diminished rapidly: v6’s reranking discovery was the single largest improvement (+0.041). By v7, automated metrics had plateaued. v8’s contribution was not in composite improvement but in adding a new dimension (AR) that revealed maxtok_768 as measurably more helpful despite near-identical composite scores.

Key Findings Across All Versions

1. Sentence_window_1 is the optimal chunking strategy for ExpertQA with this model. It maximizes fidelity (adherence, NLI) while maintaining adequate coverage.
2. Reranking hurts on ExpertQA. Confirmed in both v5 and v6 across multiple window sizes. The cross-encoder filters out relevant chunks in this multi-domain dataset. This finding may not generalize to narrow-domain datasets.
3. Binary threshold metrics have a hard ceiling problem. When most responses exceed a quality threshold, the metric saturates and can no longer discriminate. Continuous formulations are essential for high-performing systems.
4. Composite metrics plateau before quality converges. The top-4 configs are equivalent on automated metrics but differ on AR. This suggests that automated metrics capture necessary but insufficient quality dimensions.
5. max_tokens=768 is a sweet spot. More generation budget (768 vs the 512 default) produces measurably more helpful answers (AR d=0.221 vs anchor, p=0.014). The additional generation budget plausibly allows the model to elaborate rather than truncating, though this hypothesis was not directly tested.
6. Temperature has minimal impact. temp_greedy (0.0) vs anchor_detailed (0.1) differ by just d=0.125 on AR and d=0.008 on composite. Even this small temperature gap produces negligible differences.
7. Prompt template matters more than expected. The structured template produces the most user-helpful responses (AR 4.44) but worst composite (0.552). This tension between automated and human-aligned metrics deserves future investigation.

8. Final Recommendation

Expected Performance

Latency estimates from v6 sw_1 main run (~12–13s mean generation time). Failure rates from v5–v6 observed rates across all sw_1 runs. Composite and AR from v7 re-evaluation and v8 judge scoring, respectively.

Operational Characteristics

• Latency: ~12.5s mean, ~19–20s p95 (v6 sw_1 measurements)
• Failure rate: <0.5% (v5–v6 observed across >13,000 requests)
• Throughput: ~0.175 req/s sustained (v5 measurement, single-instance)
• Resource requirements: Single RTX 4090, ~23 GiB peak VRAM (12.2 GiB model + KV cache + evaluation models)

Selection Rationale

Decision rule: Select the configuration that ranks in the top tier on automated composite metrics AND performs best on answer relevance, excluding any configuration with a large automated-metric deficit.

maxtok_768 is the recommended configuration under this rule:

1. Composite: Rank #2 (0.582), within 0.001 of the best (topk_3, 0.583). The difference is not statistically significant (p > 0.8).
2. Answer Relevance: Rank #2 (4.365), significantly better than anchor_detailed (#6, d=0.221, p=0.014) and topk_3 (#7, d=0.218, p=0.013).
3. No other config ranks top-2 on both dimensions: topk_3 is #1 composite but #7 AR. temp_greedy is #3 on both. structured is #1 AR but #8 composite.
4. max_tokens=768 is the differentiator: The only parameter change from the anchor config (512 tokens) is +256 max tokens. This extra generation budget produces measurably more helpful responses (d=0.221 on AR) at negligible composite cost.

Caveats

• All results are specific to ExpertQA with Nemotron-9B-NVFP4. Generalization to other datasets or models requires validation.
• Answer relevance calibration showed moderate agreement (kappa=0.469). The relative ranking is more trustworthy than absolute scores.
• The optimization used one-factor-at-a-time (OFAT) sweeps, not factorial designs. Interaction effects may exist between parameters that were not explored.

9. Limitations

1. Single dataset: All results are from ExpertQA only. Findings like “reranking hurts” may not generalize to narrow-domain datasets where v3 (covidqa) and v4 (techqa) showed reranking benefits.
2. Single model: Nemotron-Nano-9B-v2-NVFP4 is a specific quantized model. Larger or differently quantized models may shift optimal configurations, particularly for max_tokens and prompt template effects.
3. OFAT sweep design: Parameters were swept one at a time. Factorial designs would reveal interaction effects (e.g., window_size × reranking, temperature × max_tokens) but at exponentially higher cost.
4. Moderate calibration kappa: The AR judge achieved kappa=0.469 (moderate), below the 0.6 threshold for substantial agreement. While 88% within-1 agreement and significant Spearman correlation support relative ranking reliability, absolute AR scores should be interpreted cautiously.
5. Metric reformulation mid-study: Switching from binary to continuous metrics between v6 and v7 means composite scores are not directly comparable across that boundary. The composite scale shifted from ~0.77–0.81 (binary) to ~0.55–0.58 (continuous).
6. No large-scale human evaluation: The AR judge is an automated proxy for human judgment. The 50-sample calibration set may not capture the full distribution of human preferences. A larger human evaluation (N > 200) would strengthen confidence in AR-based conclusions.
7. Completeness metric history: Completeness showed zero discriminative power under binary formulation in v5 (F=1.77, p=0.15) and was flagged for removal. However, the v7 continuous reformulation restored strong discrimination (Friedman χ²=785.22, η²=0.175, the strongest of all metrics), justifying the weight increase from 0.10 to 0.20. The metric’s utility is formulation-dependent.
8. Non-independent test cases: The same 150 test cases were used across all versions and configurations. While paired analysis accounts for within-subject correlation, there is no guarantee these 150 cases represent the full distribution of real-world queries. Performance on unseen questions may differ.

10. Artifact Index

Analysis Reports

Key Data Files

Design Documentation

Appendix A: Answer Relevance Judge Protocol

Judge Model

• Model: Qwen3.5-9B (BF16 precision)
• Inference engine: vLLM with enforce_eager=True, max_model_len=4096, gpu_memory_utilization=0.85
• Temperature: 0.0 (deterministic scoring)
• Max output tokens: 1024
• Thinking mode: Disabled (enable_thinking: False)

Scoring Scale

Evaluation Criteria

1. Does the response directly answer the user’s question?
2. Does the response use the provided context effectively?
3. Is the response appropriately detailed (not too brief, not excessively verbose)?
4. Would a domain expert find this response useful?

Additional scoring rules instruct the judge not to reward verbosity, to use the full 1–5 scale (reserving 5 for truly excellent responses), and to score 3 for on-topic but gapped responses.

Prompt and Rubric

The exact judge prompt is defined in scripts/rag_benchmark_v8/config.py (JUDGE_USER_TEMPLATE). The system prompt instructs the model to act as an expert answer quality judge and output valid JSON only. The user prompt provides the question, retrieved context chunks (numbered), and the response to evaluate, followed by the rubric and scoring rules.

Output format: {"score": <1-5>, "reasoning": "<brief justification>"}. Parsing uses strict JSON first, then regex fallback. Parse rate: 100% across all 1,598 responses.

Blinding and Ordering

• Blinding: The judge sees question, context chunks, and response only: no configuration identity, run name, or parameter values are provided.
• Presentation order: Responses are scored sequentially in CSV row order (one response per judge call). No within-response randomization is applicable since each call evaluates a single response.
• Position bias: Not applicable in this protocol (single-response evaluation, not pairwise comparison).

Calibration

50 human-labeled samples scored by the judge before full deployment. Results: kappa=0.469 (moderate), MAE=0.580, within-1 agreement=88%, Spearman rho=0.509. See scripts/rag_benchmark_v8/calibrate_judge.py for the calibration protocol.

Known Biases and Limitations

• Verbosity bias: Possible but not measured. Longer responses may receive higher scores from the judge regardless of quality. Anti-verbosity instructions are included in the prompt, but their effectiveness was not independently validated.
• Rubric iteration: The rubric was revised once after initial calibration (iteration 1 showed judge bias toward score 5). The deployed rubric includes explicit anti-inflation instructions.
• Kappa below threshold: Weighted kappa (0.469) fell below the 0.6 target for substantial agreement. Relative ranking is more robust than absolute scores.