Self-Hosting Open-Source LLMs in the EU: A Practical GDPR Compliance Checklist

Title

Self-Hosting Open-Source LLMs in the EU: A Practical GDPR Compliance Checklist

Meta Description

A hands-on guide for EU teams: self-host open-weight LLMs within the EU using Ollama to sidestep GDPR cross-border transfer headaches, with Docker Compose deployment code, a hardware sizing table, and a 10-point compliance checklist.

Article Content

Sending prompts to a cloud LLM API is, at bottom, a cross-border transfer of personal data. GDPR requires every transfer to have a legal basis and a documented purpose, and when the servers sit outside the EEA, you also need a Chapter V transfer mechanism. Rather than layering compliance paperwork onto every API call, run open-weight models on infrastructure you control, located within the EU. This article provides a runnable deployment setup plus a companion compliance checklist—the very points your legal team will press you on.

Why self-hosting changes the GDPR equation

When you call a third-party AI API, the provider is a processor acting on your instructions: you need a Data Processing Agreement (DPA), you must verify storage locations and the sub-processor list, and any transfer out of the EEA requires an adequacy decision or Standard Contractual Clauses. The EU-US Data Privacy Framework adequacy decision currently remains in force, upheld by the EU General Court in September 2025, but an appeal to the Court of Justice is still pending—two previous frameworks have already been struck down. Planning for change is prudence, not alarmism.

Self-hosting open-weight models eliminates the hardest problems at the architectural level: no external processor sees your prompts, so there is no DPA, no sub-processor list, and no international transfer in the inference traffic; no vendor training risk—paid API tiers already promise not to train on API data, and owning the model makes the question moot entirely; data residency holds by default—a server in Falkenstein or Roubaix means the data is physically in the EU, with no contractual gymnastics required.

The honest trade-off: you bear the operational and security burden of the infrastructure, and you remain the data controller. DPIA, legal basis, and AI Act classification all still apply. Self-hosting is not a compliance cheat code; it just shrinks the blast radius.

The controller-processor distinction is the starting point of GDPR data governance. The controller determines the purposes and means of processing; the processor handles data only on the controller's behalf. When you call a third-party API, the provider is the processor, and you need a DPA to constrain its behavior; when self-hosting, you play both controller and processor, directly bearing full compliance responsibility. This distinction determines how obligations are allocated: the controller is responsible for legal basis, DPIA, and transparency, while the processor is responsible for security safeguards and assistance duties. Self-hosting eliminates the external processor link but does not eliminate controller obligations—which is exactly why the article stresses that "self-hosting is not a compliance cheat code." Understanding this is essential to correctly assessing the compliance benefits and residual risks that self-hosting brings.

The stack that works in 2026

The current Ollama version is v0.40.3, with the official image supporting NVIDIA CUDA, AMD ROCm, and a built-in Vulkan runtime; every model the CLI can serve simultaneously exposes a REST API and provides an OpenAI-compatible endpoint at /v1.

On the model side, the most natural starting point for EU teams is Mistral's open-weight models—an EU company, which even removes the transfer question for the weights themselves. Llama 4, Qwen 3, and Gemma 3 are also in the Ollama model library. Verify the license for every model you plan to deploy: "open weight" does not equal "open source," and commercial terms vary widely.

For hardware within the EU, dedicated servers from Hetzner (Nuremberg/Falkenstein), OVHcloud, and Scaleway are the usual choices. Rough sizing guidance for 4-bit quantized models:

Model size	Required RAM/VRAM	Realistic hardware
8B (Q4_K_M)	~6 GB	Any modern VPS or laptop CPU
32B (Q4_K_M)	~20 GB	High-memory dedicated server
70B (Q4_K_M)	~40–48 GB	GPU server (e.g., Hetzner's RTX PRO 6000 Blackwell machines)

Starting with an 8B model on CPU is fine—it is good enough for the tasks where self-hosting pays off most: summarization, classification, and structured extraction. Scale up only when latency or quality genuinely becomes a blocker.

4-bit quantization (e.g., Q4_K_M) is a model compression technique that reduces each weight parameter from a 16-bit floating-point number to roughly 4-bit integer representation, cutting VRAM usage by about 70–75% with usually manageable precision loss. The "K" in Q4_K_M refers to the k-quants quantization method, and "M" stands for medium compression level. For most inference tasks, the output quality difference between 4-bit quantized models and full-precision versions is small, while memory requirements drop substantially, allowing a 70B model to run on a single GPU with 48 GB of VRAM. Note when deploying: quantized models still require the original tokenizer and configuration files, which Ollama handles automatically on pull; different quantization levels (Q2, Q4, Q8) trade off quality against speed, and for production it is recommended to start with Q4_K_M and upgrade only when quality falls short.

Get a deployment running in fifteen minutes

The Docker Compose below puts Ollama's model data on a persistent volume, with the GPU section commented out by default—uncomment it only on GPU hosts:

services:
  ollama:
    image: ollama/ollama:0.40.3
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"   # bind to loopback only; expose externally via reverse proxy
    volumes:
      - ollama_data:/root/.ollama
    # GPU variant: uncomment on CUDA hosts
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    environment:
      - OLLAMA_HOST=0.0.0.0
volumes:
  ollama_data:

Pull the model and verify the API:

docker compose up -d
docker exec ollama ollama pull mistral:7b-instruct
curl -s http://localhost:11434/api/tags | head -c 300
curl -s http://localhost:11434/v1/models | head -c 300

The second call hits the OpenAI-compatible endpoint, so existing application code only needs a base URL change:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="mistral:7b-instruct",
    messages=[{"role": "user", "content": "Extract the sender name from this email: ..."}],
    max_tokens=64,
)
print(resp.choices[0].message.content)

Before connecting real data: the port above is bound to loopback, so any external access must go through a reverse proxy with TLS and authentication. An unauthenticated LLM endpoint is a data breach waiting to happen—and under GDPR, that breach is yours.

The GDPR compliance checklist, item by item

1. Document your legal basis in writing

Most internal tools rely on legitimate interest, and the balancing test should be completed and documented before launch. For employee-facing systems, consent is almost always the wrong choice; contract performance covers only what the contract genuinely requires.

Legitimate interest is one of the legal bases under GDPR Article 6(1)(f), applicable when processing is necessary for a legitimate business purpose and does not override the rights of data subjects. When relying on it, a three-step balancing test is required: identify the legitimate interest (e.g., internal efficiency gains), demonstrate the necessity of processing (no less intrusive alternative), and balance against data subjects' reasonable expectations and impact. The conclusion of the test must be documented in writing and included in the privacy notice. For employee-monitoring systems, legitimate interest is usually more reliable than consent—because consent is rarely freely given in an employment relationship. Note: legitimate interest does not apply to public authorities exercising their official functions, and data subjects have the right to object, so an easy objection channel must be provided.

2. Data Protection Impact Assessment (DPIA)

Run a DPIA before launch, especially when outputs influence decisions about people or when sensitive data is processed at scale. Do it first—the answers to most of the remaining items get written down here.

3. Data minimization in prompt construction

Strip out what the model doesn't need before you build the prompt: if one field suffices, don't send the whole customer record; if a redacted summary works, don't stuff in the raw email thread. Minimization must be implemented in the code that assembles prompts—it is a technical control, not a policy statement.

4. Retention and deletion—including your own logs

Prompts, conversation history, and logs containing personal data are themselves personal data. Set retention periods and enforce them automatically—a nightly cron is worth more than a policy document:

# Purge inference logs older than 30 days (adjust to your policy)
0 3 * * * psql "$DATABASE_URL" -c \
  "DELETE FROM inference_logs WHERE created_at < NOW() - INTERVAL '30 days';"

Deletion requests must reach this store. When a user asks for erasure, logs, history, and caches all need to be covered—"the model weights don't remember it" is not an answer.

5. Access control and encryption

TLS in transit, encryption at rest for the model data volume, credential rotation. Even with no vendor to point fingers at, the Article 32 security obligations are not reduced one bit.

6. Audit logs—log access, not content

Record who queried the system and when, and which model responded. Think twice before logging full prompt content: content logs are a second copy of personal data, carrying their own retention and deletion obligations. Logging metadata or hashes is often sufficient.

7. Incident response

A personal data breach on the LLM host starts the 72-hour notification clock to the supervisory authority. Have a runbook ready: who revokes credentials, who assesses the scope of impact, who handles notifications. There is no one else's security team watching this machine.

8. Transparency

Your privacy notice must state the AI system and where processing takes place. "Data never leaves our EU servers" is a strong and truthful statement for this architecture—say it, and keep it true.

9. Automated decision-making

If outputs affect significant decisions about individuals on a solely automated basis, GDPR's automated decision-making safeguards apply. In most cases the fix is adding a human review step—which is also the fix for model errors.

10. EU AI Act deployer obligations

The AI Act is now fully in force: deployers bear transparency obligations—informing people when they are interacting with AI-generated content—as well as incident reporting and system technical documentation duties.

AI Act deployer obligations refer to the responsibilities borne by the deployer of a high-risk AI system (i.e., the organization using it), as distinct from the provider's (developer's) obligations. Deployers must ensure human oversight, monitor system operation, retain automatically generated logs for at least 6 months, conduct a fundamental rights impact assessment in certain cases, and inform affected persons about the system's use. For most internal tools built on self-hosted LLMs, if they do not fall into a high-risk classification (e.g., not used in employment, education, or critical infrastructure scenarios), the main obligations are transparency disclosure and incident reporting. But the classification must be assessed yourself: if LLM outputs directly influence credit, hiring, or judicial decisions, high-risk obligations may be triggered and should be evaluated as part of the DPIA.

What self-hosting does not solve

• Training data risk. The EDPB links unlawfully processed training data to deployment risk. A model trained on scraped personal data may memorize it, and as the deployer you inherit that exposure. Ask how the model was trained and write the answer into your DPIA.
• Output risk. Hallucinated personal data about real people is still personal data you have processed. Human review of consequential outputs is a compliance control, not just a quality measure.
• Licensing. Weight licenses vary; some restrict commercial use or redistribution. "Open weight" is a technical fact, not a legal one.
• Cost. Cloud APIs bill per token; self-hosting bills monthly plus your time. At low volume, APIs are usually cheaper. The justification for self-hosting lies in compliance and data control, not the invoice—run the numbers against your own traffic first.

What you can start doing this week

Spin up the CPU compose file on an EU host, run one real workflow through an 8B model, and then write the DPIA based on what you learned. The technical part takes an afternoon; the documentation part takes a week. This order—a working system first, then walking the checklist against a real system—is what keeps the paperwork honest. Self-hosting alone won't make you compliant, but it removes the questions you can't answer, leaving only the ones you should have been answering all along.

Multi-source corroboration

The conclusions above cross-check across several independent sources: dev.to's enterprise LLM guide and promptquorum.com's local LLM privacy manifesto both support, from an architectural angle, the judgment that "self-hosting eliminates the Article 44 transfer problem"; the compliance checklists from januscompliance.co.uk and vdf.ai jointly emphasize that self-hosting does not change controller status and that DPIA and legal basis remain your responsibility—these two points form the article's core argument. Hardware and deployment parameters come from bitdoze.com's Ollama configuration guide and qovery.com's hands-on notes on Hetzner GPU machines; the time-sensitive judgment about the pending Data Privacy Framework appeal relies on the single source cited in the original, and readers should defer to the latest announcements from supervisory authorities.

Note: this article is a compliance preparation checklist from an engineering perspective and does not constitute legal advice.

One-line Summary

Self-hosting open-source LLMs eliminates GDPR cross-border transfer and processor headaches, but controller obligations remain your own to fulfill.

Paragraph Summary

The article examines the GDPR compliance issues raised by sending prompts to cloud LLM APIs and proposes self-hosting open-weight models within the EU as an alternative. Self-hosting removes three categories of risk—external processors, international transfers, and vendor training—but the organization remains the data controller, so obligations such as DPIA and legal basis are unchanged. It provides a Docker Compose deployment based on Ollama v0.40.3, covering model selection, hardware configuration, and OpenAI-compatible endpoint integration, and lists a ten-point compliance checklist: legal basis, DPIA, data minimization, retention and deletion, access control and encryption, audit logs, incident response, transparency, automated decision-making, and AI Act deployer obligations. It closes by noting that training data risk, output hallucination, and license restrictions cannot be eliminated by self-hosting, and that cloud APIs may be more economical at low volume.