Search Function
Share article

Making corporate knowledge usable with GenAI and Retrieval Augmented Generation

Technical contribution

August 19, 2024
futuristic image that symbolically represents artificial intelligence in the form of a brain with data points

"How do I have to adjust a component in my machine of a certain series to reduce vibrations?"

"What is the best way to repair it if a fault occurs?"

"Where can I find more information on how to solve a problem sustainably?"

In the past, one of our customers had to wade through mountains of documents to find answers to such questions, investing a lot of time and nerves. Today, he poses these questions to an interactive chatbot, quickly receives the right answer and can turn his attention to the actual solution to the problem just as quickly.

The second article in our AI Deep Dive series looks at how artificial intelligence can help companies make the most of their in-house expertise and the technologies and steps required to do so.

Innovative knowledge management thanks to generative AI and RAG

A chatbot like the one we described in the introduction is made possible by a technology that is currently attracting a great deal of media interest: Generative Artificial Intelligence (GenAI) or Large Language Models (LLM). Made famous by applications such as ChatGPT and Microsoft Copilot, these can give very eloquent answers based on publicly available information and have been incorporated into the training of the models behind them. The use case described, however, is aimed at internal company knowledge that has not been trained into the models. In such cases, Retrieval Augmented Generation (RAG) provides a remedy. RAG expands the capabilities of LLMs by using ad hoc external data in addition to the trained "knowledge" to answer user questions. In the scenario described above, the chatbot uses machine documentation, maintenance reports, error lists, etc. as external data, processes them and incorporates the information they contain into its answer. In this way, linguistic interfaces to internal company data can be created relatively easily and quickly. Fine-tuning the LLMs, which is extremely resource-intensive and requires huge amounts of data, is therefore unnecessary.

Data preparation: the key to successful AI application

Data preparation is essential for a functioning RAG application. In order for an LLM to process the texts well, they must be cut into suitably sized sections, enriched with metadata and embedded by LLMs (embedding: a numerical representation of the semantic meaning of a text).

The requirements for data pre-processing are very individual depending on the customer situation, as they depend on many factors such as the source formats, the quantity or consistency of the data. Due to successful implementations and the resulting increase in expectations of RAG applications, we are also experiencing an ever-increasing range of data to be integrated within individual customer projects.

Another challenge when integrating your own data into RAG applications lies in the internal structure of the documents. Typical documentation texts contain not only plain text, but also many illustrations, such as diagrams, drawings or photos, tables, cross-references, warnings and the like. In addition, many documents are full of abbreviations and cryptic terms that are initially incomprehensible to outsiders (and therefore also to LLMs).

Everything contains important information, and often the meaning only becomes apparent when all elements of a document are considered together, which makes editing essential. For example, illustrations can be verbalized, abbreviations written out, cryptic designations explained in short texts and important metadata such as warnings saved. And all of this must ultimately be automated and performed efficiently.

Four steps to optimal data processing

In order to tackle this challenge in a structured way and to be able to implement efficient processing routes quickly, we roughly divide the data processing process for RAG applications into four steps: Import, Cracking, Enrichment and Provisioning. We will provide a brief overview below.

 A diagram of a process Description automatically generated

  1.  Import: Provision of the source data at an accessible location

In this step, the output data to be processed must be provided in a known and accessible location. Depending on the output format, this can be in an object memory or a database, for example.

  1. Cracking: Structuring and extraction of texts, tables and figures

In the cracking section, the source data is broken down and fundamentally structured in order to prepare it for further processing. Essentially, this involves recognizing texts, tables and illustrations of all kinds as such in order to be able to treat them separately in subsequent steps.

The effort required here can vary greatly. Structured or semi-structured formats often only require simple parsing scripts to process the data. For highly unstructured formats such as PDF files, the process is more complex. Here, it may be necessary to extract the content using complex machine learning-based processes. Examples of this include optical character recognition (OCR) or layout analysis based on computer vision.

It is not only important to extract content such as images, tables and text, but also to include contextual information. This can be, for example, image captions or the position of an element in the document hierarchy ((sub)headings).

  1. Enrichment: Enrichment of the data for use by RAG applications

In the enrichment step, the data is enriched and shaped in such a way that it can later be meaningfully used by RAG applications. This step depends on the type of data and can be as complex as required.

  • Text: Essentially, texts should be divided into content-related sections (segmentation), which can then be enriched with metadata. It can be useful, for example, to classify the content of a section or to identify specific text objects (entities) in order to be able to filter for certain topics later or to programmatically display warnings.
  • Tables: Basically the same applies to tables, except that they must first be converted into a form that can be processed by an LLM. This can be a textual description or a format such as Markdown, for example.
  • Images: There are two ways to make images accessible for RAG applications. On the one hand, they can be embedded as such and thus, like texts, transferred to a vector space that represents their meaning. Secondly, they can be verbalized using multi-modal LLMs such as GPT4-Vision. The description obtained in this way is then treated like normal text.

The result of the enrichment step is content-encapsulated and coherent segments that are enriched with the most meaningful metadata possible.

  1. Provisioning: Preparation and provision of the enriched segments

Finally, the enriched segments are prepared and made available for the use of a specific LLM. The first step here is chunking. This involves cutting the segments into a size that is suitable for the respective RAG application, which primarily depends on the token limit of the LLM used there. The segments are further subdivided or combined accordingly.

The individual chunks are then embedded and made available in a vector database to which the RAG application has access. It makes sense to keep references to the source data so that the RAG application can refer to them later when generating the response. In addition to the vector database, there are other ways of providing the data, e.g. via graphs, database tables or similar.

If large volumes of data are to be processed and these grow over time, it can make sense to design the process described incrementally for reasons of efficiency in order to avoid redundant processing.

Implementing data pipelines: challenges and solutions

Whether incremental or not, the implementation of data pipelines that cover all steps can be quite a complex undertaking, depending on the initial situation. Suitable tools can help here. When making a selection, it is important to remember that there is no one solution that can do everything and always fits. A careful analysis of the initial situation is therefore essential.

Thanks to our broad portfolio, we can flexibly select and implement the most suitable technologies and tools for our customers. For the customer mentioned at the beginning of the article, for example, we implemented incremental pipelines on the Microsoft Azure platform with services such as Azure AI Search, Azure OpenAI and Azure Databricks, which fit in well with the customer's existing system landscape.

If you would also like to test the possibilities of LLM-based RAG applications for your use case or are already convinced and looking for an implementation partner, we look forward to hearing from you.

We recommend:

Share article

Further technical articles

[data_hub_count]
technical contribution
technical contribution
technical contribution

Name

Title
Profile

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.

Areas of Expertise
  • Focus 1
  • Focus Area 2
  • Focus Area 3
  • Focus Area 4
Placeholder for portrait photo
Do you have questions about Dataciders or our services?

Omair Ahmed Khan

Omair Khan is a technology expert specializing in data analytics, artificial intelligence, and large-scale business transformations. He currently serves as Executive Vice President of Data Analytics & AI at Swisscom. With more than two decades of experience building high-performing, data-driven organizations, he previously held senior leadership roles at Deutsche Telekom, DHL, and Infosys, where he drove cloud migrations, modernized legacy systems, and oversaw the large-scale adoption of generative AI. In addition to his operational leadership role, Omair Khan is active as an angel investor in GenAI startups and serves on several advisory boards in the fields of private equity, quantum AI, sovereign AI, and dental & vision AI. He is also the author of the recently published book “Hope It Helps!” and was named to the “Young Elite – Top 40 Under 40” list by the business magazine Capital in 2020.
Portrait of Omair Khan, Advisory Board Member at Dataciders
"
In AI, context is everything - without it, even the best models fail to deliver impact. Dataciders brings years of proven experience executing Data & AI projects across the DACH region, making them the ideal partner to scale AI's real value.

Matthias Wilcken

Matthias Wilcken is a Senior Partner at Rivean Capital and a member of Dataciders’ Advisory Board. As a member of Rivean’s Executive and Investment Committees, he is responsible for the firm’s investment activities in the DACH region. Matthias has many years of experience working with mid-sized companies. He has successfully supported numerous Rivean portfolio companies in the DACH region and across Europe in their growth, international expansion, and strategic development. In addition to Dataciders, Matthias is currently responsible, among other things, for Rivean’s investments in Engelmann, Perbility, Green Mobility, and ]init[ AG. Matthias has been part of the Rivean Capital team since 2012. Prior to that, he worked at Intermediate Capital Group and at Goldman Sachs International in London.
Profile of Matthias Wilken, Advisory Board Member at Dataciders
"
Dataciders is one of the leading providers when it comes to turning data and AI into tangible value for businesses. What matters is measurable impact through better and faster decisions and solutions that prove their value in practice. This is exactly the position we want to strengthen and expand together

Prof. Dr. Burkhard Schwenker

Prof. Dr. Burkhard Schwenker, long-standing CEO and Chairman of Roland Berger, currently serves on numerous supervisory and advisory boards, including as Chairman of the Supervisory Board of HASPA AG and Warburg Bank, as well as on the advisory boards of Init and Dataciders. He teaches Strategic Management at HHL Leipzig Graduate School of Management and serves as Academic Co-Director of the Center for Scenario Planning. Burkhard Schwenker is also actively engaged in social and public affairs, including as Chairman of the Board of Trustees of the ZEIT Foundation. A renowned author, he regularly publishes articles and books, including "Erfolgsfaktor Betriebswirtschaftslehre".
Portrait of Prof. Dr. Schwenker, Advisory Board Member at Dataciders
"
Artificial intelligence is radically transforming business models. The companies that succeed will be those that consistently align strategy, leadership, and AI while putting people at the center.

Dr. Gero Presser

CEO
Profile

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.

Areas of Expertise
  • Focus 1
  • Focus Area 2
  • Focus Area 3
  • Focus Area 4
Portrait of Gero Presser from Dataciders
Do you have questions about Dataciders or our services?

Information on Hyperautomation with n8n for End-to-End Processes

Information on the Microsoft CSP License Assessment

About Agentic Engineering Transformation

Information About the AI.Assessment

Information About the AI.Revolution Framework

Information about AI.Strategy

Schedule an in-person appointment

Request a free, comprehensive comparison of procurement channels

Information on Getting Started with AI Using Microsoft Copilot

Explore AI.Foundation

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials

Download Information Materials