Jev for RAG: Relevance Decisions and Answer Evaluation
A decision component can help assess or route retrieved material. It does not by itself establish that retrieval found the right documents or that a generated answer is faithful to them.
Locate the decision in the pipeline
| Stage | Example requirement |
|---|---|
| Retrieval | Find the potentially relevant source passages. |
| Selection | Choose or score material against a clear question. |
| Generation | Write an answer from the permitted evidence. |
| Verification | Check whether claims follow the cited passages. |
Make relevance a usable rubric
Define what a passage must contain to answer the user’s actual question. A document about the same topic may still lack the required date, product version or jurisdiction. Use clear examples of irrelevant, partially useful and directly answering material before evaluating a scoring component.
Build a diagnostic evaluation
- Use questions with known source passages and some questions with no supported answer.
- Preserve document versions and access permissions.
- Compare the retrieved set before and after any scoring step.
- Inspect whether a useful passage was incorrectly removed.
- Evaluate the final answer separately from the relevance score.
Understand the score
TypeSafe documents a Score primitive and its returned values. Read that field definition before treating an output as a probability of factual correctness. A rubric score can help order candidates without proving that the selected passage is sufficient evidence.
Compare end-to-end cost
Include retrieval, decision calls, context preparation and generation. Filtering may reduce downstream input but can also discard the evidence needed for a correct answer. Measure accepted answers and unsupported claims, not merely the amount of context removed.
A practical stopping rule
Adopt the extra step when a held-out evaluation shows a meaningful improvement in answer quality or cost at an acceptable loss of relevant evidence. Keep a route for no-answer and uncertain cases. Avoid presenting an answer as grounded merely because one intermediate score was high.
Use the blank model/decision evaluation sheet to retain model route, input, expected outcome, decision, fallback, latency and cost. No benchmark results are included.
Sources and review scope
Public-source review checked October 19, 2026. The procedures below are proposed evaluations, not experiments performed by this site.