Businesses face a difficult choice: use powerful cloud AI services that process data on external servers, or run AI locally and maintain full data sovereignty. This guide explains how to set up private, compliant AI for business use.

Why Businesses Are Moving to Local AI

The compliance case for local AI has never been stronger:

GDPR Article 44 prohibits transferring EU personal data to third countries without adequate protection — sending employee or customer data to US-based AI APIs creates risk
HIPAA prohibits sharing protected health information (PHI) with services that haven't signed a Business Associate Agreement (BAA)
Legal privilege — law firms risk breaking attorney-client privilege by sending client documents to cloud AI
Trade secrets — confidential R&D, financial projections, or competitive intelligence shouldn't leave your network
Regulatory trend — France's CNIL, Germany's BSI, and the EU AI Act all push toward on-premise processing for sensitive data

Regulatory Framework Overview

Under GDPR, you are the data controller. Using a cloud AI API makes that provider a data processor — requiring a Data Processing Agreement (DPA). Key requirements:

Requirement	Cloud AI Risk	Local AI Solution
Data minimization	Hard to guarantee with opaque APIs	Full control — process only what you need
Right to erasure	Provider may retain training data	Data never leaves your premises
Data transfer restrictions	US providers may not comply	No transfer, no restriction
Processing records	API logs may be incomplete	Full audit logs on your own systems
Breach notification	You depend on provider's detection	You control all monitoring

HIPAA Requirements

For healthcare organizations, local AI avoids the complex BAA process:

No need to negotiate BAAs with AI providers
PHI stays within your HIPAA-compliant infrastructure
Audit trails under your control
Incident response fully managed internally

Practical: What Data Should Never Go to Cloud AI

shell

HIGH RISK (always use local AI):
- Patient records, diagnoses, treatment plans
- Employee personal information, HR files, salary data
- Client/customer PII (names, addresses, IDs)
- Legal documents, contracts, privileged communications
- Financial statements, unpublished earnings, M&A plans
- Source code with proprietary algorithms
- Authentication credentials, API keys

MEDIUM RISK (evaluate case by case):
- Anonymized business data
- Internal process documentation
- Non-sensitive customer support templates
- Public-facing marketing content

LOW RISK (cloud AI generally acceptable):
- Editing publicly available content
- General research on public topics
- Grammar/style checking of non-sensitive text

Architecture Options

Option 1: On-Premise Server (Best Control)

A dedicated server in your office or data center running Ollama with an authentication layer:

Hardware recommendations:

Small team (1–10 users): Workstation with RTX 4090 (24GB VRAM) — handles 32B models
Medium team (10–50 users): Dual RTX 4090 or A6000 — handles 70B models, multiple concurrent users
Large team (50+ users): Multi-GPU server (4x A6000 or 8x RTX 4090) — vLLM for production serving

Software stack:

bash

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Set Ollama to listen on LAN only (not public internet)
export OLLAMA_HOST=192.168.1.100:11434

# Start Ollama
ollama serve

# Add authentication layer (nginx + basic auth or OAuth)
# See security configuration below

Option 2: Private Cloud (VPC/Virtual Private Network)

For teams using cloud infrastructure but needing data sovereignty:

Deploy on AWS GovCloud or Azure Government (HIPAA/FedRAMP compliant regions)
Use dedicated instances (not shared) for complete isolation
Enable VPC-only networking — no internet egress for AI traffic
Use customer-managed encryption keys (CMEK)

This is the recommended path for larger enterprises that already have cloud infrastructure.

Option 3: Hybrid (Local + Private Cloud)

The most flexible approach:

Sensitive workloads → on-premise local AI
Development/testing → private cloud
Public-facing chatbots → carefully scoped cloud AI with DPA

Securing Your Local AI Server

On managed Linux fleets, Secure Boot is usually mandatory and usually the reason the GPU driver will not load: enrol a Machine Owner Key rather than disabling it, because turning Secure Boot off to load one driver removes boot-chain verification for the whole machine.

Network Security

nginx

# nginx reverse proxy with authentication
server {
    listen 443 ssl;
    server_name ai.company.internal;

    ssl_certificate /etc/ssl/company.crt;
    ssl_certificate_key /etc/ssl/company.key;

    # Restrict to internal network only
    allow 192.168.0.0/16;
    deny all;

    location / {
        auth_request /auth;
        proxy_pass http://localhost:11434;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;
    }

    location /auth {
        proxy_pass http://localhost:8080/validate;
        proxy_pass_request_body off;
    }
}

Authentication with OAuth/SSO

For enterprise environments, integrate with your existing identity provider (Okta, Azure AD, Google Workspace):

bash

# Use Open WebUI with SSO support
docker run -d \
  -p 3000:8080 \
  -e WEBUI_AUTH=true \
  -e OAUTH_PROVIDER_NAME="Azure AD" \
  -e OAUTH_CLIENT_ID="your-client-id" \
  -e OAUTH_CLIENT_SECRET="your-client-secret" \
  -e OPENID_PROVIDER_URL="https://login.microsoftonline.com/tenant-id/v2.0" \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Audit Logging

For compliance, log all AI interactions:

python

import logging
import json
from datetime import datetime

# Audit log every request
def log_ai_interaction(user_id: str, model: str, prompt_hash: str, response_length: int):
    audit_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "user_id": user_id,
        "model": model,
        "prompt_hash": prompt_hash,  # hash, not the actual prompt (privacy)
        "response_tokens": response_length,
        "ip_address": request.remote_addr  # log source IP
    }
    logging.getLogger("audit").info(json.dumps(audit_entry))

Cost Analysis: Local vs Cloud for Business

For a team of 20 developers using AI assistance 4 hours/day:

Approach	Monthly Cost	Annual Cost	Data Risk
ChatGPT Enterprise	$25/user × 20 = $500	$6,000	Data sent to OpenAI
Claude Team	$25/user × 20 = $500	$6,000	Data sent to Anthropic
Local: RTX 4090 server	Hardware: $1,600 + $30 electricity	~$360/yr after year 1	Zero — stays on-premise
Local: Dual 4090 server	Hardware: $3,200 + $60 electricity	~$720/yr after year 1	Zero — stays on-premise

Break-even: A single RTX 4090 server pays for itself in under 3 months vs a 20-person ChatGPT Enterprise subscription.

Recommended Models for Business Use

Use Case	Recommended Model	Why
General business assistant	Llama 3.3 70B Q4	Best overall quality for business tasks
Code review & debugging	DeepSeek R1 32B or Qwen 2.5-Coder 32B	Specialized for code
Document summarization	Gemma 3 27B Q4	Long context, efficient
Customer support drafts	Qwen 2.5 14B	Fast, cost-efficient
Legal document analysis	Llama 3.3 70B Q4	Best reasoning, stays on-premise
Medical record summaries	Llama 3.3 70B Q4 (with HIPAA setup)	Quality + full data control

Licence matrix for business use

Read the licence before the leaderboard. Most open models on this site ship under Apache-2.0 or MIT and can go into a commercial product with nothing more than the licence notice. A large minority run under a vendor's own terms, which allow commercial use but add conditions: an acceptable-use policy, an attribution rule, or a cap on company size. A handful forbid commercial use outright. The table counts every model with a page in our library by the licence on its record, so it changes when the library does.

Licence	Kind	Commercial use	Models	Licence text
Apache-2.0	Permissive	yes, on its terms	148	one text for every model
MIT	Permissive	yes, on its terms	33	one text for every model
OpenMDW-1.1	Permissive	yes, on its terms	12	one text for every model
Modified MIT	Permissive	yes, on its terms	4	MiniMax M2.5 230B, Kimi K2.5, Kimi K2.6, Kimi K2.7 Code
Llama Community	Open-weight, with conditions	yes, on its terms	10	each model's card (10 models)
Gemma Terms	Open-weight, with conditions	yes, on its terms	7	one text for every model
LFM Open License v1.0	Open-weight, with conditions	yes, on its terms	7	one text for every model
Custom Open-Weight	Open-weight, with conditions	yes, on its terms	6	MiMo-V2-Pro 1T+, Qwen3.8-Max, Nex-N2.5 mini, Nex-N2.5 Pro, Nex-N2.5 Max, GLM-5.3
BigCode OpenRAIL-M	Open-weight, with conditions	yes, on its terms	3	one text for every model
NVIDIA Open Model License	Open-weight, with conditions	yes, on its terms	3	one text for every model
Llama 3.2 Community	Open-weight, with conditions	yes, on its terms	2	Llama 3.2 Vision 11B, Llama 3.2 Vision 90B
MiniMax Community	Open-weight, with conditions	yes, on its terms	2	MiniMax M3 428B-A23B, MiniMax M3.1 Flash 428B-A23B
Yi License	Open-weight, with conditions	yes, on its terms	2	Yi 1.5 34B Chat, Yi 1.5 9B Chat
Kimi K3.1-Pro License	Open-weight, with conditions	yes, on its terms	1	Kimi K3.1 Pro
Kimi Open-Weight	Open-weight, with conditions	yes, on its terms	1	Kimi K3 2.8T
MiniCPM Model License	Open-weight, with conditions	yes, on its terms	1	MiniCPM-V 4.5
Mistral Ecosystem Open-Weight	Open-weight, with conditions	yes, on its terms	1	Mistral Large 4 1T-A52B
Mistral Modified MIT	Open-weight, with conditions	yes, on its terms	1	Mistral Medium 3.5 128B
Muse Model License	Open-weight, with conditions	yes, on its terms	1	Muse Spark Open 816B
Qwen Community License 1.0	Open-weight, with conditions	yes, on its terms	1	Qwen3.8-Flash-Next
StepFun Model License 1.0	Open-weight, with conditions	yes, on its terms	1	Step 5 Preview 600B-A27B
Upstage Solar License	Open-weight, with conditions	yes, on its terms	1	Solar Open 2 250B-A15B
CC-BY-NC-4.0	Non-commercial / research	no	7	one text for every model
EXAONE AI Model License 1.2 - NC	Non-commercial / research	no	3	EXAONE 4.5 33B, EXAONE 4.0 32B, EXAONE 4.0 1.2B
Mistral Research	Non-commercial / research	no	2	Ministral 3B, Ministral 8B
MNPL-0.1	Non-commercial / research	no	1	one text for every model
Not stated in the record	—	check the model card	2	Tev1 4B, Tev1 0.8B

263 models with a page in the library, grouped by the licence on their record. "Licence text" links the one canonical text where a licence has one; otherwise each named model's page links its own card, because a vendor's own agreement or a modified MIT differs from model to model. This is information, not legal advice: read the licence text before you ship. What each kind means in practice:

Permissive (Apache-2.0, MIT, BSD, OpenMDW). Ship it, and keep the licence notice with the weights. Apache-2.0 also grants a patent licence from every contributor, which MIT does not spell out. That is one reason legal teams tend to prefer it.
Modified MIT. The MIT text plus a vendor clause, usually an attribution duty once a product passes a size threshold. The threshold differs from model to model, so read the card of the exact model you deploy; the table links each one.
Open-weight with conditions (Llama, Gemma, NVIDIA, Qwen Community, MiniMax and others). Commercial use is allowed, but the terms bind you to an acceptable-use policy and sometimes to a cap. Liquid AI's LFM Open License, for example, ends commercial rights once your company's annual revenue passes the threshold its text states, after which you need a paid licence. These are contracts you accept by using the weights, not notices you can ignore.
Non-commercial (CC-BY-NC-4.0, Mistral Research, MNPL). Fine for evaluation and research. You cannot build a product on them. If one of these tops your internal benchmark, look for a permissively licensed model one row down rather than hoping nobody asks.
Not stated. A model whose publisher has not stated terms is not "free to use". Treat it as unlicensed until the publisher says otherwise.

A quantised re-upload does not change the licence. A GGUF on Hugging Face is uploaded by whoever converted it, and the licence shown on that repo is whatever that person wrote into the card's metadata. The weights are still a derivative of the original. Gemma's terms, for one, require anyone who redistributes a model derivative to pass on the agreement and its use restrictions. So check the base model's licence, not the re-upload's tag, and keep both repo URLs in your records.

What I keep on file for every model a business deploys:

The upstream repo, the exact revision or file hash, and the licence text as it stood that day.
Whether the licence caps users or revenue, and where your company sits against the cap.
Any attribution the licence requires, and where your product shows it.
A re-check whenever you upgrade. Licences change between versions of the same family: compare Gemma 3 27B with the Gemma 4 family, which moved to Apache-2.0.

This section is information, not legal advice. The licence text, linked in the table, is the document that binds you.

Sources, checked 2026-10-14: Apache-2.0 §3, patent grant · LFM Open License, commercial-use threshold · Hugging Face: a repo licence is set in its card metadata · Gemma Terms of Use §3.1, redistributing derivatives.

Implementation Checklist

shell

LEGAL / COMPLIANCE
□ Document your AI use cases and data flows
□ Update Privacy Policy to reflect local AI use
□ Create AI Acceptable Use Policy for employees
□ If EU users: confirm no personal data leaves EU network
□ If HIPAA: confirm PHI stays within HIPAA-compliant environment

TECHNICAL
□ Deploy Ollama on dedicated hardware or private cloud
□ Configure network to block external AI API calls
□ Set up authentication (SSO/OAuth preferred)
□ Enable audit logging (who used AI, when, what model)
□ Configure backups for model storage
□ Document incident response plan for AI-related breaches

OPERATIONAL
□ Train employees on approved AI tools and data classification
□ Establish model update process (when to upgrade models)
□ Set up monitoring for hardware health (GPU temps, VRAM usage)
□ Create vendor assessment process if using cloud AI for any use case

Frequently Asked Questions

Do I need a DPA with Ollama? No. Ollama is open-source software you run yourself. There's no external data processor involved.

Can we use open-weight models commercially? Most popular models (Llama 3, Gemma 3, Mistral, Qwen 2.5) allow commercial use. Check the specific license: Llama 3 has a community license allowing commercial use for companies under 700M monthly active users.

Is a GGUF from a community uploader under a different licence than the original model? No. The quantised file is a derivative of the original weights, so the original licence still governs it, whatever the re-upload's card says. Check the base model's licence (see the licence matrix above).

How do we handle model updates? Pull new model versions with ollama pull model:version. Test in staging before deploying to production. Document each model version used for auditability.

Is local AI good enough to replace cloud AI for business? For most business tasks (writing, summarization, code review, data analysis), a local 70B model is competitive with GPT-4-class performance. For complex reasoning or very long context (>100k tokens), cloud APIs still have an edge — but the gap is closing with each new model release.

Next →Cybersecurity Operations