LLM evaluation before and after go-live | Anexia

For the AI quality review, we build a task set with traceable origins. It contains typical cases, difficult exceptions and tasks that are deliberately unsolvable. Expected results are described so that several domain experts can assess them consistently. Where there is no clear reference answer, well-founded assessment criteria are needed. A second model can help with this, but should not take over the entire quality decision unchecked.

When carrying out LLM tests, we separate development material from the set used for the final assessment. If every adjustment is optimised on the same examples, the result can overestimate how well the application works in everyday use. LLM evaluation therefore also uses new or withheld tasks. Measurement results remain linked to the model version, prompt, data source and configuration. The effort of human review is recorded as well. A model with slightly better answer quality can be economically unsuitable if waiting times or rework rise significantly. The conclusion of the model evaluation therefore shows differences per task class and the limits of the test set, instead of deriving a general performance promise from a single metric.

Scope of services for LLM evaluation

The agreed scope of the AI quality review can include the following results: test dataset, assessment rubric, reference answers, evaluation process and regression tests for model changes. For each key result, we define what it is for, who reviews it and in what form it is handed over. This allows development, operations and procurement to assess the same service. Vague umbrella terms are replaced with concrete work packages in the offer.

Carrying out LLM tests also includes coordinating the necessary interfaces with your team. We clarify access, contact persons and available testing options before the relevant implementation steps. Changes to the scope are described together with their impact on effort and sequence. This gives you a clear overview of delivered results, outstanding prerequisites and additional requests.

The key technical decision in LLM evaluation

Automated assessment models can pre-sort results, but they are checked for systematic misjudgements against cases reviewed by humans. For the model evaluation, we document the relevant assumptions and explain the chosen approach. Key alternatives are considered against the same functional requirements. This makes later adjustments easier when load, data or organisational responsibility change.

A sound decision also takes maintainability and handover into account. In the AI quality review, we check which skills are available internally and what additional support is needed. This connects technical possibilities with a realistic operating model. The client can see which tasks will recur regularly after the project and which decisions remain with them.

A test set that is constantly used for optimisation loses its meaningfulness for unknown cases. We treat this point as a concrete review question when carrying out LLM tests. A risk is linked to the affected workflow, its possible impact and a suitable measure. Where prerequisites are still missing, the statement explicitly remains open until the necessary investigation has taken place.

We separate development and acceptance data and look at quality scores by case type rather than only as a single overall average. For LLM evaluation, we record the starting point, the tested configuration and the result. Acceptance refers to the agreed scope. Open restrictions are documented visibly and assigned to a next step with clear ownership. This allows your team to distinguish between proven functionality, a deliberate decision and an assumption that is not yet confirmed.

Project example as an illustrative scenario

The following scenario is an application example and not a customer reference. An internal assistant answers questions about company policies. Individual demonstrations seem convincing, but different phrasings of the same question lead to different answers. LLM evaluation would therefore first create a test set assessed by domain experts. It would contain clear questions as well as outdated information, contradictory sources and cases with no permissible answer.

For the model evaluation, assessment rules would be defined before models or prompts are compared. A linguistically fluent answer could be incomplete in substance; a correct statement could still use an impermissible source. These dimensions would need to be recorded separately. Domain reviewers would resolve disputed assessments together. Automated evaluations could complement the review, but would have to be examined for their meaningfulness against examples assessed by humans. Changes would be tested with inputs that are as comparable as possible.

The release after the AI quality review would include known error classes and remaining uncertainties. An average score must not hide critical individual errors. The plan would include a versioned test set, traceable assessment rules and an evaluation for each relevant user group or task class. After go-live, new error cases could be added to the review. On this basis, it would be possible to judge whether a model change helps overall or whether it worsens an important use case.

Our experience and our engineering approach

We combine custom software development with the technical tasks of introduction and further development. For the model evaluation, we bring the perspectives of application, integration and operations together in a shared workflow. The functional requirements remain connected to the technical decisions. You work with a defined scope of services and traceable results.

Our experience should be concretely verifiable for your project. That is why, in a technical exchange, we explain how we investigate the relevant risks, which work results are planned and how your team can assess them. For the AI quality review, we align the roles involved with actual needs. Available references are placed in the context of the respective project and company; confidential customer details are not used as general marketing claims.

Effort for the AI quality review

The effort depends on domain complexity, the availability of qualified reviewers and the coverage of rare cases with serious consequences. We therefore calculate LLM evaluation on the basis of a clearly defined scope and explicitly stated prerequisites. An initial analysis step can make sense if key technical information is still missing. After that, work packages and acceptance criteria can be agreed more reliably.

The model evaluation can be organised as a limited project, as a joint implementation with your team or with subsequent support. Rights of use, access, documentation and responsibilities are clarified in the contract. Ongoing operating hours, response times and external consumption costs are covered by separate agreements. This makes it clear which services are provided once and which tasks recur regularly after introduction.

Scope limits of LLM evaluation

A passed evaluation applies to the tested configuration and does not replace ongoing monitoring after data or model changes. When carrying out LLM tests, we record this limit in the offer itself. Any additional services required are included with their functional justification and are not silently assumed. Your team can therefore plan suitable internal skills and divide the overall work sensibly between the parties involved.