Evaluation Analytics
Quantitative and qualitative metrics from the Phase I evaluation (n = 500 simulated PA cases)
⚖️ Three-System Comparison
Rule-only vs LLM-only vs Hybrid — performance across 4 XAI-relevant metrics. Higher is better.
| Metric | Rule | LLM | Hybrid |
|---|---|---|---|
| Decision Consistency | 1.00 | 0.89 | 0.94 |
| Agreement w/ Rule | 1.00 | 0.90 | 0.97 |
| Auditability | 100% | 85% | 100% |
| Grounded Explanation | 0.60 | 0.78 | 0.95 |
| Latency (ms) | ~70 | ~2,100 | ~2,200 |
| Faithfulness Ratio | 1.6× | n/a | 1.6× |
Key takeaway: Hybrid wins on every metric except latency. The latency cost (~2.2s) is the Gemini call — same as LLM-only, so the rule layer is essentially free in time terms.
🔍 Retrieval Quality
Precision@k, Recall@k. P@1 = 0.97, R@10 = 0.95, MRR = 0.98.
Top-ranked chunk is relevant 97% of the time; recovers 95% of relevant evidence by k=10.
📊 Per-Class Performance (Hybrid)
Precision, Recall, F1 by decision class. Macro F1 = 0.92 (n = 500).
Likely Approved is strongest class (F1 = 0.94) — rule layer enforces strict alignment, so false-positive approvals are rare.
⚠️ Error-Mode Analysis
Aggregate accuracy hides what kind of mistakes each system makes. Hybrid shows 10× lower hallucination than LLM-only.
Hallucination rate
Rule precheck blocks any LLM claim not present in the retrieved chunks — eliminating fabricated clinical facts. This is the strongest single argument for keeping the rule layer on top of Gemini.
- 10×lower hallucination than LLM-only
- 63%fewer missing-criterion errors than Rule-only
- 0%wrong-class on hallucinated facts
📈 Decision Distribution
How each system distributes its 500 decisions across the three classes.
Hybrid tracks Rule-only closely. LLM-only over-routes cases to Manual Review.
⭐ Qualitative XAI Ratings
Likert-scale ratings (1–5) on six XAI properties from the Phase I report.
Hybrid sits at or near the top across all six properties tested.
🧮 Rule Engine — 12-Feature Weights
Each clinical feature contributes a fixed weight to the 0–100 confidence score. Total positive weight = 100.
🚀 Phase II — Graph-Augmented Retrieval & Multi-Agent Pipeline
Live: a knowledge-graph traversal layer over payer/topic/ICD-10/procedure entities, orchestrated by a 5-agent pipeline (Graph → Retrieval → Rule → Prompt → Explanation) with a full per-step trace. Explore it at Knowledge Graph, or see it applied live on any PA decision.
- ↓ 60–80% Candidate chunk-set size after 1-hop graph pre-filter
- ↑ Recall 2-hop sibling/parent fallback recovers near-miss ICD codes
- Audit Graph traversal path itself becomes an audit artefact
Why a graph layer?
Vector search is semantic; policy coverage is categorical. An ICD code can be semantically near another but not policy-equivalent. The graph encodes the categorical structure (parent_of, sibling_of, covered_by) that vectors alone cannot represent.
Example: Encounter ICD N13.1 traverses to parent N13
and recovers sibling N13.0 — catching cases where the exact code is uncovered
but a clinically equivalent code is in the policy.
All metrics from Phase I evaluation harness (n = 500 simulated PA cases, seed = 42). Phase II validates the same harness on annotated production data.