RAG Evaluation Consulting

With a golden set of real questions and verified answers, then separate metrics: retrieval hit@k, answer faithfulness to retrieved context, citation correctness, and behavior on empty retrieval. Run those in CI against the last pin. Evaluate as the asking user so ACL is in the suite. That is RAG evaluation consulting in practice, delivered in four weeks in your cloud.

A versioned list of questions drawn from real use, each with gold passages, a gold answer or refusal, and the user identity that should be allowed to see them. Fifty-plus is a starting point, not a trophy. Identifier questions and empty-retrieval cases must be in it or hybrid retrieval and refuse-when-empty will never be tested. You own the set.

We use whatever scorer we can pin and reproduce in your CI. LLM-as-judge is allowed if the judge prompt and model are versioned and you pay inference. Unpinned vendor scores are not a gate. Retrieval hit@k does not need a judge. Faithfulness and citations often do. Model-agnostic: we can swap the judge the same way we swap the generator.

Retrieval: hit@k and the hybrid-vs-vector delta. Generation: faithfulness to retrieved context, citation coverage and correctness, refusal on empty retrieval. Operations: ACL leak tests. Leaderboard scores that ignore your corpus and permissions do not matter. We report by query class so identifier failures cannot hide inside a blended average.

Four weeks standard to a CI-wired harness and a first golden set, then 30 days on-call. Labeling capacity is the usual constraint. Financial services typically 8–12 weeks when validation packages are in scope. Healthcare typically 10–14. Ecommerce typically 6–10. Scope is fixed in writing.

Yes. Week 1 is an audit of what you measure today. Most teams have a demo notebook and no CI gate. We keep your stack, add the golden set and gates in the client cloud, and hand over IP. We do not require a rebuild unless retrieval or ACL cannot be tested as-is.

You do. Client owns IP: questions, labels, harness, pins, runbooks, in the client cloud. No shared training on your questions. Thirty days on-call for failing gates. After that, label expansion is a new fixed scope. We are not a hosted eval SaaS you cannot export.

Fixed-scope implementation fee plus platform fee, quoted after we see question volume and whether judges run on every pull request. You pay inference. No token markup. If you only need a labeling guide, we will say so rather than selling a four-week harness you will not run.