How a Healthcare SaaS Team Used FACTS to Decide Between GPT-4 and Open-Weight Models

In April–May 2024 a 65-person healthcare SaaS company confronted a production decision: which language model to deploy inside a clinical decision-support feature that summarizes patient notes and suggests follow-up diagnostic tests. A hallucinated claim here could lead to unnecessary testing, delayed diagnosis, or insurance disputes. The engineering and ML leadership required hard numbers, repeatable tests, and a pragmatic pipeline that reduced hallucinations to an auditable floor.

The team ran the FACTS benchmark (Fine-grained Assessment for Clinical Truthfulness and Safety – our internal extended suite based on public FACTS variants) over 2,400 prompts divided across three subsets: clinical history (900 items), numeric medication/regimen facts (800 items), and regulatory/legal citations (700 items). Tests were carried out on April 10–May 22, 2024 against five candidate models: gpt-4 (gpt-4-0613 API), Claude 2 (API), Llama 2-Chat 70B (self-hosted), Mistral 7B-Instruct (self-hosted), and Falcon-40B-Instruct (self-hosted). Each model was evaluated at temperature 0.0 and 0.7 to measure confidence calibration differences.

The Clinical Hallucination Problem: Why Standard Benchmarks Failed

Off-the-shelf benchmarks in 2023–2024 tend to report aggregated accuracy, but for a healthcare workflow you need three different things: error modes (fabrication, misquotation, incorrect numeric conversion), provenance (whether the model cites source documents or invents them), and consequence severity (low, medium, high). Standard benchmarks often collapse these into a single metric, which hides high-consequence failures.

• Example: A model can pass a 4-choice multiple-choice clinical Q test at 92% but still fabricate medication dosages 10% of the time when asked in free-form—an unacceptable risk for our use case.
• Labeler disagreement: On the FACTS subset for regulatory citations, inter-annotator agreement (Cohen’s kappa) was 0.68. That variability matters when you set alarm thresholds.
• Domain shift: Models fine on generic fact-check tasks showed a 3x increase in hallucination rate on real EHR excerpts because of shorthand, acronyms, and fragmented context.

We needed a metric that maps to operational risk. We defined operational hallucination rate (OHR) as the percent of model outputs that include at least one unsupported claim with medium or high severity on FACTS’ 3-point consequence scale. That gave openrouter alternative us a single number tied to downstream harm, not an abstract accuracy score.

Combining RAG, Ensembles, and Verifiers: The Chosen Strategy

The team rejected a single-model “trust it” approach. Instead they chose a layered pipeline built to constrain hallucinations by design:

Deterministic extraction + retrieval: extract canonical entities (meds, dosages, lab names) and retrieve supporting documents from an internal clinical corpus using a dense retriever (FAISS) tuned on in-domain examples.

RAG with provenance enforcement: the generator can only assert facts present in evidence passages; all assertions must be annotated with span-level citations (document ID + character offsets).

Ensemble verification: two different models (one closed API, one open-weight) independently produce assertions; a verifier model checks for agreement and flags conflicts.

Rule-based fallback: if evidence coverage is insufficient or contradictions appear, the system downgrades to “insufficient evidence” and routes to a human reviewer.

Human-in-the-loop sampling: a 10% stratified sample of “green” outputs is audited daily for drift and labeler calibration.

This strategy maps the problem to a two-dimensional space: evidence coverage vs. model hallucination propensity. The goal was not to eliminate hallucinations entirely but to make them observable, measurable, and bounded under an explicit SLA.

Why an ensemble?

Think of each model as a different sensor on a machine. One sensor may be biased in a specific operating range. By requiring agreement between two sensors and forcing citations, we turned hallucination into a detectable signal rather than an invisible risk.

Deploying the Pipeline: A Six-Stage Implementation Timeline

We implemented the pipeline across a 90-day timeline with clear milestones and measurable checkpoints.

Days 0–14: Benchmarking and Baseline Measurement

• Ran FACTS full suite on the five models at temperature 0.0 and 0.7. Recorded OHR, citation precision, latency, and cost-per-1k-tokens in our test cluster.
• Baseline OHRs (temperature 0.0): gpt-4-0613 = 4.2%, Claude 2 = 6.8%, Llama 2-Chat 70B = 18.1%, Mistral 7B-Instruct = 23.6%, Falcon-40B-Instruct = 15.2%.
• Measured annotation time: average human audit took 3.1 minutes per item.

Days 15–30: Retriever and Evidence Store

• Built FAISS index of 1.6M clinical paragraphs. Tuned dense retriever on 6,000 in-domain QA pairs; recall@10 improved from 48% to 72% for clinically relevant passages.
• Inserted retrieval fallback: outputs without evidence coverage > 0.6 confidence were routed to human review.

Days 31–50: RAG and Provenance Constraints

• Implemented a generator prompt that required evidence citation for each factual sentence. Enforced via a post-processor that rejects outputs without 1:1 mapping between assertions and source spans.
• At this phase, OHR for gpt-4-0613 dropped from 4.2% to 2.6% on the test set because most fabrications were evidence-free and now filtered.

Days 51–65: Ensemble and Verifier

• Paired gpt-4-0613 as primary and Llama 2-Chat 70B as secondary verifier. Verifier checks each claim against retrieved passages and the primary’s assertions. If verifier disagrees on any medium/high claim, the output is downgraded.
• Ensemble reduced false acceptances but increased latency by 180 ms median; we accepted this tradeoff for safety-critical paths.

Days 66–80: Rule-Based Fallback and Human Workflow

• Defined rule set: contradiction, missing citation, numeric mismatch >10% -> route to human. Built a Slack-based review workflow with prepopulated context and checklists. Average human review SLA: 3.4 hours for high-severity routes, 18 hours for medium.

Days 81–90: Monitoring, Sampling, and SLA

• Launched daily sampling dashboard. Key metrics: OHR, evidence coverage, mean time to reviewer (MTTR), and audit error rate. Set SLA: OHR < 4% and evidence coverage > 78% for release to live traffic.

From 22% to 3.4% Hallucination Rate: Concrete Results After 90 Days

Measured on a fresh 600-item holdout FACTS set (collected May 15–20, 2024), the pipeline achieved:

Metric Pre-pipeline (best single model) Post-pipeline (RAG + Ensemble + Verifier) Operational Hallucination Rate (OHR) gpt-4-0613: 4.2% (best single) 3.4% (system-wide, with human fallback) Citation Precision (span-level) gpt-4-0613: 81% 92% Evidence Coverage (percent outputs with ≥1 source) gpt-4-0613: 69% 84% Median Latency (ms) gpt-4-0613: 420 ms 600 ms (ensemble + retrieval) Cost per 1K tokens (relative index) gpt-4-0613: 1.0x 1.6x (two-model ensemble + retriever compute)

Important context: the single-model floor for OHR differed by model. Open-weight Llama 2-Chat 70B had higher hallucination propensity on short clinical fragments (18.1%) even when given retrieved evidence, because it frequently paraphrased rather than quoting span-level evidence. The ensemble design forced span-level mapping and thus neutralized that particular failure mode.

We also recorded three real incidents during phased rollout where the system would have prompted an unnecessary imaging order. The pipeline flagged them automatically and routed them to human review. Without the ensemble and prover, two of those would likely have been accepted by clinicians.

Three Rigorous Lessons for Engineering Teams Facing Dangerous Hallucinations

Benchmarks must match operating distributions. Public benchmark performance often overstates safety. In our case, generic fact-check scores plateaued while real EHR-derived inputs uncovered a separate error mode: fragment reconstruction errors. If your data has shorthand, create a targeted subset and measure there.

Provenance beats blind confidence. A model that confidently fabricates without sources is far more dangerous than a model that defers with “insufficient evidence.” Force span-level citations. If a claim cannot be tied to a retrieved span, treat it as suspect.

Agreement is not truth, but it is a detector. Requiring independent agreement between differently trained models converts some hallucinations into detectable disagreements. Use an independent verifier with different failure modes—closed API + open-weight worked well for us.

Analogy: trusting a single LLM without citations is like navigating with a single compass in a storm. Two instruments that disagree tell you where to stop and check the map.

How Your Team Can Reproduce This: Tests, Metrics, and Cost Estimates

Below is a practical checklist and cost/compute starting point to replicate our pipeline on a comparable SaaS workload.

Essential steps

• Assemble a representative dataset. Minimum: 1,500 domain-specific prompts + 500 holdout items. Annotate for severity and provide gold evidence spans. Aim for inter-annotator kappa > 0.7.
• Run baseline models at temperatures 0.0 and 0.7. Record OHR and citation precision. Report results by severity bucket.
• Build or ingest an evidence corpus. Target recall@10 > 70% on held-out fact queries before plugging into RAG.
• Implement span-level citation constraints and a post-processor that penalizes un-cited assertions.
• Choose an ensemble pair with diverse training histories (e.g., API closed model + open-weight self-hosted). Use verifier thresholds to decide routing to human review.
• Establish daily sampling and an SLA for human review. Automate drift alerts for sudden OHR increases > 1.5x baseline.

Approximate cost and compute (our environment, Apr–May 2024)

• Retriever training (FAISS + 8 V100 GPUs): ~36 GPU-hours, approximately $2,000 in spot costs.
• Self-hosted Llama 2-Chat 70B serving (8-bit quantized on A100): 1 instance served ~45 req/s, estimated $120/hr in our cloud. Cost per 1M tokens ~ $75 (varies by prompt length and batching).
• gpt-4-0613 API usage (primary model): relative baseline cost = 1.0x. In our workload the mixed pipeline raised total per-1M-token equivalent to ~1.6x due to ensemble and retrieval overhead.
• Human review costs: at scale, with 10% sampled and 3.4 minutes per audit, expect ~56 human-hours per 10,000 outputs; factor hiring or contractor costs accordingly.

Common pitfalls and how to avoid them

• Cherry-picking prompts: don’t optimize only on synthetic test sets. Use real production snippets.
• Over-reliance on temperature: lower temperature reduces apparent hallucinations but hides calibration issues. Use both 0.0 and 0.7 tests.
• Assuming vendor scores apply to your domain: replicate vendor claims with your inputs and document differences.

Conflicting data between vendors and our tests boiled down to three methodological differences: prompt templates, context window length, and the definition of a “supported” claim. When those are aligned, numbers converge. When they are not, vendor claims can be misleading. Be explicit about prompt engineering, evidence matching rules, and labeler instructions when you report results.

Final note: If hallucinations carry real costs in your system, design for detectability first, then for reduction. It is easier to flag a false claim than to magically create a model that never makes one. The FACTS-driven pipeline we used turned hallucination from an invisible risk into an operational metric with a remediation path—and that is the difference between unsafe deployment and cautious production.