Search Function
Technical contribution
Imagine this: your invoices are processed automatically, contracts are analyzed, and customer inquiries are categorized—all without a single document ever leaving your company network. Sound like a pipe dream? It’s not.
Many companies face the same challenge: AI-powered assistants promise massive efficiency gains, but compliance requirements such as the General Data Protection Regulation (GDPR) often prohibit the use of cloud-based services. This is no minor issue: 48% of German companies cite data protection as a key barrier to AI adoption (Bitkom 2025)—while at the same time, 485% more corporate data is flowing into AI tools than just one year ago (Cyberhaven 2024 ).
The solution isn’t in the cloud, but right here on your premises. The combination of local AI platforms—such as Ollama or LMStudio for prototype development, and vLLM or Xinference for production use—and frameworks like LangChain enables GDPR-compliant AI integration that runs entirely on-premises.
In this article, you'll learn …
The use of on-premises LLMs is becoming increasingly important, as companies not only need to comply with data protection regulations but also want to retain control over their data. In addition, using on-premises LLMs offers cost advantages over expensive cloud services.
The GDPR requires companies to maintain control over personal data and to be able to fully document its processing. When using external APIs such as OpenAI, Claude, or Google Gemini, the following challenges arise, among others:
On-premises AI fundamentally solves these problems: Data never leaves the company’s infrastructure, there is no reliance on the cloud, and there is complete control over the entire processing workflow—a crucial advantage not only for GDPR compliance but also with regard to the EU AI Act.
In addition to compliance, on-premises LLMs also offer economic advantages—especially for companies with high, consistent processing volumes. Those who process thousands of documents daily benefit from predictable fixed costs rather than variable API fees. For smaller companies or scenarios with fluctuating workloads, however, a cloud model may be more flexible and cost-effective—in these cases, it’s worth conducting a customized comparison. The key advantages at a glance:
By using local LLMs, you can save costs in the long run by avoiding expensive cloud services while gaining greater flexibility in how you use your resources.
LangChain is an open-source framework that significantly simplifies the integration of large language models (LLMs) into applications by providing abstract interfaces and reusable building blocks for common use cases. Building on this concept, LangChain4j brings these capabilities to the Java ecosystem, enabling the seamless integration of LLMs into Java applications. Compared to direct API calls, it offers several key advantages.
In traditional LLM integrations, LLM responses are often treated as free-form text that must subsequently be processed, validated, and checked for errors manually. This approach is difficult to maintain in enterprise applications, vulnerable to changes in prompts, and offers little resilience to model changes or unexpected response formats. Type-safe return formats offer a decisive advantage here in terms of stability, maintainability, and testability.
Instead of working with unstructured raw text, LangChain4j allows you to directly extract LLM responses into typed Java objects. A declarative interface clearly defines what data is expected and the structure in which it should be returned:
Based on the return class (“Invoice”), the framework automatically generates a JSON schema and directs the LLM to produce a strictly structured output. The result is validatable, type-safe data that integrates seamlessly into existing Java architectures—without error-prone free-text parsing, regex strings, or implicit assumptions about the model’s response format.
LangChain4j abstracts the specific LLM provider through a unified API. This allows for the interchangeable integration of different LLM providers, such as OpenAI, Anthropic, and Azure OpenAI Service, as well as local inference solutions like Ollama, LMStudio, vLLM, and others.
Switching providers typically requires only a configuration adjustment, not changes to the application code. This specifically reduces the risk of vendor lock-in and makes it possible to switch between cloud-based and on-premises models depending on the use case. This flexibility is particularly crucial in the context of the GDPR, for example, to process sensitive data on-premises and use cloud models only where it is not a regulatory concern.
How exactly can you establish a connection to your application-specific Java data classes in your enterprise application?
Langchain4j offers a handy feature for this: @Description annotations, which are attached directly to data class fields:
These descriptions automatically become part of the JSON schema and help the LLM interpret fields correctly—especially in the case of multilingual documents or fields with synonyms. However, it is not necessary to explicitly define every possible language or all conceivable synonyms. Rather, the annotations serve as targeted additional cues that assist the language model in classification. Modern LLMs already possess a broad understanding of language, so that often just a few representative examples are sufficient to reliably improve classification without creating maintenance or scaling issues.
In summary, LangChain4j handles the translation between structured document JSON and typed business objects. The framework:
The following examples demonstrate how local LLMs support specific business processes—from document processing to content analysis.
Challenge: Invoices come in various formats, from different suppliers, and are sometimes in multiple languages. Employees have to laboriously enter the relevant information into the appropriate software manually.
LLM Solution: The model understands semantic-spatial relationships by utilizing positional data associated with various text blocks. This spatial information is extracted, for example, by an upstream PDF parser. For example, “Rechnung Nr.,” “Invoice No.,” and “Bill #” are automatically recognized as synonyms, and positional data helps the LLM with classification—a value to the right of “Gesamtbetrag:” or “Total:” is highly likely to be the total amount. When multiple dates are present, the model determines the due date based on context.
KPI: Processing time of less than one minute per invoice. Data extraction occurs automatically in the background. A human-in-the-loop can review and correct the results.
The following screenshots show a sample PDF, a prompt, and a comparison of extraction results for Gemma3 1B vs. Gemma3 4B.
Sample PDF and prompt
Model-Result Comparison
The results suggest that even small models are capable of extracting relevant data.
Challenge: Legal documents contain complex clauses, cross-references, and implicit risks that can only be identified by understanding the overall context.
LLM solution: The model extracts key clauses such as notice periods, liability limitations, and SLAs, identifies unusual or high-risk wording, and automatically summarizes the results in a structured format.
Compliance benefit: Contract details (which are often highly sensitive) never leave the company. The on-premises model can also handle M&A documents or patent applications without the risk of external disclosure.
KPI: Initial risk assessment of a standard contract in seconds instead of hours—no need for a manual preliminary review.
Challenge: Incoming emails or tickets must be categorized, prioritized, and forwarded to the appropriate teams.
LLM Solution: The model classifies incoming messages by category (complaint, inquiry, order, etc.), assesses sentiment to prioritize urgent cases, and extracts structured information such as customer ID or the product in question. The LLM step can be integrated directly into the data pipeline—between the input channel and the CRM system.
KPI: No more manual pre-sorting— routing decisions are made in a matter of seconds. This gives employees more time to handle time-consuming inquiries.
The model ecosystem is evolving rapidly—new, more powerful open-source models are released almost every month, which can be run locally or integrated into custom applications. Many of these are available under open licenses (e.g., Apache 2.0), which makes it easier to use them in commercial projects as well.
The following table provides an example of the RAM requirements for the Gemma 4 as a reference; comparable values apply to similarly sized models from other families (data is based on official Google specifications):
Smaller variants (E2B, E4B) also run on non-GPU hardware and are suitable for high-volume, time-sensitive applications. Larger models (31B, 26B, A4B) offer higher accuracy for complex tasks such as contract analysis. Modern quantization (8-bit, 4-bit) significantly reduces memory requirements with minimal loss of accuracy.
To compare how well such open-source language models perform in certain tasks compared to proprietary competitors, public websites such as https://llm-stats.com/ are a good resource.
In the context of AI, “inference” refers to the execution of a trained AI model—that is, the process of generating specific answers or predictions from input data—and it is precisely this step that places special demands on performance, scalability, and operations. An inference platform provides the technical runtime environment in which models are efficiently executed—it handles, among other things,model serving, resource management, parallelization, scaling, and the provision of an API for client applications.
Ollama has proven itself for prototyping and internal development, thanks to its easy installation and large community. For high-throughput production use, vLLM is the ideal choice (Apache 2.0, multi-GPU support, continuous batching). If you need a cluster-ready solution with a supervisor/worker architecture and an integrable vLLM backend, Xinference is a good fit.
The tools mentioned differ primarily in terms of scalability and operating model, and each offers different inference engines and runtime optimizations. Since they provide an OpenAI-compatible API, it is easy to switch between them within the client application depending on the project phase—in Langchain4j, this requires nothing more than a configuration change.
The development of local AI is advancing rapidly, making it even easier to get started in the future. Three trends are particularly relevant in this regard:
The integration of local LLMs into business processes is no longer just a vision for the future. They offer a real competitive advantage, particularly for companies with strict compliance requirements, sensitive data, or high processing volumes. With the right framework and models, this can already be achieved today:
Are you interested in integrating AI assistants into your processes?
Our team will guide you every step of the way, from the initial feasibility analysis through to full-scale operation—in compliance with the GDPR, practical, and tailored to your specific use case:
Contact us for a no-obligation initial consultation—we’ll show you how to integrate AI into your processes without compromising your data.
Sources:
Mariano Frohnmaier is a Senior Consultant at Dataciders with a background in AI and natural language processing. After spending several years adapting pre-trained language models for specific domains and testing them in practice, his focus today is on automating business processes with Axon Ivy—and on integrating AI effectively into real-world business processes.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.
“Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.”
