RAG development for reliable enterprise search | Anexia

A knowledge-based answer can contain two different errors. The search finds the wrong excerpt, or the model draws a wrong conclusion from a correct excerpt. Each error needs a different correction. We therefore evaluate retrieval and answer generation separately. A better-worded prompt does not repair an index in which relevant documents are missing or important version relationships have been lost.

Tables, embedded figures and documents with similar product names are particularly demanding. A meaningful test dataset therefore contains questions where the answer has to distinguish between two almost identical documents. Deleted or locked documents are checked as well, because their content must not remain findable through old index entries. This also applies to metadata and cached answers. Acceptance therefore covers the entire update path, starting at the source system and ending at the visible source reference. Only then can you judge whether the solution can be reliably maintained in a knowledge environment that changes constantly.

The first technical decision concerns the origin and usability of the information. Developing RAG systems requires a clear answer as to which source is authoritative, who maintains it and which people may access which content. A successfully connected dataset can be unsuitable in substance if older versions sit alongside current requirements or project boundaries cannot be recognised. We therefore record update cycles, deletion events and the adoption of access rights. For a pilot, a clearly limited dataset is often sufficient, provided it actually contains the relevant difficulties. These include contradictory information, missing information and restricted visibility rights. This creates a verifiable starting point for RAG development. It prevents a supposed model problem from later having to be reworked as a data or permissions problem.

Before approval, the RAG integration needs a yardstick that the business unit and the technical team understand alike. To this end, we describe typical tasks, acceptable results and errors that must never go unnoticed. An average across very different tasks is not enough. An incorrect internal summary has different consequences than an inaccurate instruction given to a customer. The evaluation within RAG development therefore differentiates by use case and impact. Difficult cases are deliberately included and not removed from the test dataset because they worsen the metric. Correctly rejecting a request that cannot be answered also counts as a possible quality characteristic. Retrieval augmented generation thus receives approval criteria that go beyond a subjectively successful demo session and can be applied again for later changes.

Human approval in the development of RAG systems is only effective if the reviewer receives sufficient context. A finished text without sources, assumptions or recognisable changes can hardly be assessed reliably under time pressure. We therefore design the review around the actual decision. Relevant source data, uncertain passages and proposed changes are presented together. The responsible person can reject, correct or request additional information. This feedback is made usable for quality improvement without confidential content being reused in an uncontrolled way. At the same time, we measure the review effort. RAG development only creates an economic advantage when the support, including the necessary checks, enables a better process than the previous way of working.

In RAG integration, models differ in more than price and answer length. Their behaviour with incomplete information, structured outputs, processing conditions and suitability for specific technical language are also relevant. We compare variants using the same set of tasks and document the configurations used. A better general benchmark does not replace this comparison. Versions and possible changes to external services are also taken into account. Before a switch, it is clear which loss of quality is unacceptable and how the previous configuration can be used again. The decision made during the RAG integration for retrieval augmented generation therefore remains verifiable. A cheaper model can make sense if it fulfils the defined purpose; additional capabilities only justify higher costs if there is a recognisable benefit.

The cost-effectiveness of developing RAG systems cannot be derived from a price per model call. A single transaction can require several calls, longer contexts and additional checks. We therefore record costs along the business outcome and include failed attempts and rework. A cheaper configuration can turn out to be more expensive if users have to ask follow-up questions or correct results more often. Suitable limits are set for different usage patterns. These include traceable consumption budgets and controlled handling of unusual load. RAG development thus gains a cost model that supports product decisions. The analysis shows which transactions work economically and where changes to data, process or model choice make more sense than pure price optimisation.

For RAG integration, changes outside the actual application code can be decisive. New documents, different access settings or a revised prompt change the result even though no classic software version has been released. We therefore define which components belong to a verifiable state and how changes are documented. If a result looks unusual, it must remain traceable which sources and settings were involved. The information stored for this purpose is limited to the necessary extent. A suitable release process combines domain tests with technical approval. Retrieval augmented generation can thus be developed step by step without losing the link between a change and its effect. If problems arise, this traceability makes targeted correction easier.

A source alone does not prove a correct conclusion. We therefore also check whether the cited passage actually supports the specific statement and belongs to the approved version. For the RAG integration, the answer is refined on the basis of your specific environment. What matters are the verifiable prerequisites of the project.

For retrieval augmented generation, a short description of the affected process, the most important systems and known limitations are helpful. Confidential documents can be provided through a suitable channel after consultation. A complete requirements specification is not needed for the first exchange. We then state which information the technical assessment actually still requires.

For the RAG integration, the agreed results, checks and open points are documented. This includes the relevant configuration and the prerequisites for its use. The scope depends on the contract. Acceptance should make clear what was checked, which limitations exist and who takes on the remaining tasks.

For retrieval augmented generation, we limit the required information and access to the agreed purpose. Roles, data paths and the termination of access are agreed. If personal data or particularly sensitive content is involved, the responsible specialist departments are brought in. The technical implementation follows the requirements defined as a result.