Search Function
Technical contribution
For some months now, our customers have therefore been increasingly approaching us to use AI as specialized assistants for their individual requirements. For example, chatbots are being created that provide answers based on extensive technical manuals. In other applications, AI provides almost unobtrusive support by making recommendations in what was previously purely manual product documentation.
Particularly important: the AI's response must always be comprehensible and verifiable for the user.
When the user interacts with our AI implementations via the chat function, they not only receive an answer. The AI also links to the documents in which the information used for the answer was found.
The creation of a comprehensive database is important for the generative AI technologies described here. Unfortunately, this step is often underestimated for reasons of time and cost. However, the large number of texts that companies have collected in their manuals, wikis and documentation are invaluable for generative AI. Only when these texts, which often contain a large part of the company's knowledge, are made accessible to AI can their potential actually be exploited.
Large quantities of texts are unstructured data whose usefulness is not always easy to understand. Just because I have a book on the shelf doesn't mean I know what's in it and how best to use the knowledge it contains. Even a sophisticated system that ensures that the book is in the perfect place on the shelf doesn't help me.
AI systems are no different to humans in this respect. Just as a human must first read a book, take notes if necessary and relate what they have learned to what they already know, an AI must first break down a text into digestible sections, add metadata and further context. Only then can humans and AI actually do something with the text.
For AI technologies, there is the additional challenge that typical documentation texts contain not only text, but also all kinds of illustrations, tables, headings, notes, footnotes and the like. People find it easy to relate all of these to each other. However, even the best current multimodal AI models reach their limits and this challenge will remain for the foreseeable future.
The solution to these problems lies in the pre-processing of documents.
Pipelines must be implemented that obtain the documents from their original sources, transform them, relate their elements to each other, enrich them with metadata and finally store them in a new form in a location that the AI can then access (typically a search index or a vector database).
This pre-processing has a direct influence on the quality of the actual AI application: the more specifically the pipeline is adapted to the respective documents, the higher the quality and reliability. In our experience, no other adjusting screw during the development of an AI application has a comparable influence on the end result.
That's why you have to expect that 70-80% of the effort during the implementation of an AI application (without testing and evaluation) is spent on the pre-processing pipelines.
After successful data preparation and implementation of the AI application, there is another important step. The AI application must function reliably for use in day-to-day business. In particular, false statements, also known as hallucinations, must be ruled out as far as possible.
Careful preparation of the data creates an important basis for reliable application.
In any case, the application must be carefully tested before going live. Systematic testing of a solution is widespread in all areas of IT. It is therefore not surprising that proven test procedures are also needed for AI applications. However, the non-deterministic behavior of AI requires procedures that differ from the test methods used in traditional IT.
Our projects are based on three test methods.
Even though we have mainly talked about the challenges in this article, we would like to encourage you to develop your own customized AI applications. Because let's be honest: if you want to stay ahead of the competition, off-the-shelf AI solutions won't help you.
Our team will be happy to assist you in designing AI tools for your individual application and, in particular, help you with the all-important data preparation.
Dr. Karl Häfner studied economic geography and has been working as a data scientist in the field of natural language processing and AI since 2019. He has been supporting companies in integrating generative AI and large language models into their business processes since 2021. At the Dataciders subsidiary ixto GmbH manages he the AI Solutions team.
Dr. Simon Birkholz studied physics and started working as a data scientist in 2016. He has worked as a consultant for Mercedes-Benz, Siemens, Vattenfall and Bayer, among others. Today he is Managing Director of the Dataciders subsidiary ixto GmbH with a focus on data science and AI.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.
“Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.”
