- Ollama is the model runner. Everything else is application layer. You can replace Ollama with any OpenAI-compatible endpoint (OpenAI, Anthropic via LiteLLM) without touching the rest of the stack.
- LiteLLM as a central proxy means every tool in your stack points at one URL. Swap models, add rate limits, and track token costs in one place.
- n8n and Dify solve different problems: n8n automates workflows across many services; Dify builds AI applications (RAG chat, knowledge bases) as the product itself.
- GPU is only required for Ollama. All other tools run on CPU; a $40/month VPS runs the whole stack minus local inference.
- SearXNG is what lets your AI agents search the web without leaking queries to Google or Bing.
The stack at a glance
Seven tools, one coherent system. The table maps each to its role, its resource cost, and whether it requires a GPU.
| Tool | Role | GPU required | Connects to |
|---|---|---|---|
| Ollama | Local model server (LLM runtime) | Recommended; CPU fallback works | LiteLLM, Open WebUI, n8n, Dify, Agent Zero |
| Open WebUI | Chat interface (ChatGPT-like UI) | No | LiteLLM or Ollama directly |
| LiteLLM | Unified API gateway for all models | No | Everything |
| n8n | Workflow automation + AI agent node | No | LiteLLM, SearXNG, any API |
| Dify | RAG pipelines + chat app builder | No | LiteLLM or Ollama, vector DBs |
| SearXNG | Private metasearch for agent web access | No | n8n, Dify, Agent Zero |
| Agent Zero | Autonomous agent with OS access | No | LiteLLM, SearXNG, MCP tools |
A self-hosted AI stack in 2026 consists of a local model runner (Ollama), a unified API proxy (LiteLLM), a chat interface (Open WebUI), workflow automation (n8n), a RAG and app builder (Dify), a private search engine (SearXNG), and an autonomous agent framework (Agent Zero). All seven tools deploy via Docker and run on a single machine with 16 GB RAM. Only Ollama benefits from a GPU for inference; the rest are CPU-only services.
Layer 1: Ollama - the model runner
Ollama is the foundation. It downloads models from its registry and serves them via a REST API on port 11434 that is compatible with the OpenAI API format. One command installs it; ollama pull llama3.1:8b downloads an 8B model. On a machine with a GPU, models run at 50-100 tokens per second. CPU-only inference works but runs at roughly 3-8 tokens per second on a modern 8-core CPU.
The models worth running in 2026:
- Llama 3.1 8B (5 GB): the general-purpose default. Fast, capable, runs on 8 GB VRAM.
- Qwen 2.5 7B (4.7 GB): strong at code and structured output. Same VRAM tier.
- Mistral 7B (4.1 GB): fast inference, good instruction following.
- Llama 3.1 70B (40 GB): needs 48 GB+ VRAM or multi-GPU; use a cloud API instead for most self-hosters.
- Qwen2.5-VL 7B: the same model covered in the Qwen2.5-VL guide - adds vision (image understanding) to the local stack.
Full install guide, GPU troubleshooting, and the OpenAI-compatible API setup are in the Ollama setup guide.
Layer 2: LiteLLM - the API gateway
LiteLLM is an OpenAI-compatible proxy that sits in front of all your model endpoints. You point it at Ollama (local), OpenAI, Anthropic, Gemini, Mistral, or any other provider. Every other tool in your stack points at LiteLLM's single /v1/chat/completions endpoint instead of each model directly.
Why this matters in practice: when you want to switch from a local Llama model to Claude 3.5 Sonnet for a specific workflow, you change one model name in LiteLLM's config rather than reconfiguring n8n, Dify, Open WebUI, and Agent Zero individually. LiteLLM also adds per-key rate limiting, spend tracking, and a dashboard for token costs across providers.
LiteLLM acts as a unified OpenAI-compatible API proxy for all model providers. A self-hosted stack routes Ollama, OpenAI, Anthropic, and Gemini calls through one endpoint at http://localhost:4000/v1. Every application (n8n, Dify, Open WebUI, Agent Zero) uses a single API key and URL; switching models is a config change in LiteLLM, not a reconfiguration of every downstream tool. The full setup is in the LiteLLM proxy guide.
Layer 3: Open WebUI - the chat interface
Open WebUI is a self-hosted chat interface that looks and behaves like ChatGPT. It connects to Ollama directly or through LiteLLM, supports multiple users with separate conversation histories, and has a built-in model management panel. If you want to use your local models with a familiar interface and nothing else, Open WebUI is the whole story: pull the Docker image, point it at Ollama, done.
Setup guide and multi-user configuration are in the Open WebUI guide.
Layer 4a: n8n - workflow automation with AI agents
n8n is where your local AI stack connects to the rest of your tools. Its LangChain-based AI Agent node lets you drop a model (via LiteLLM), a memory backend, and tools like SearXNG or a database onto a visual canvas. Trigger examples: a webhook that summarizes incoming emails, a scheduled flow that scrapes a website and classifies results, a Slack command that queries your knowledge base.
Where n8n beats writing code: 400+ prebuilt integrations mean you connect to Slack, Notion, GitHub, Google Sheets, and hundreds of other services without writing HTTP clients. Where code beats n8n: if the logic is complex Python, consider moving it to Windmill (see the n8n vs Windmill comparison).
Full Docker setup and queue-mode production config are in the n8n self-hosted guide.
Layer 4b: Dify - RAG applications and chat builders
Dify occupies a different niche than n8n. Where n8n automates processes, Dify builds AI products: knowledge base chatbots, document Q&A apps, and multi-step chat pipelines with retrieval. Upload a PDF, configure a vector store (Dify ships with Weaviate and others via Docker Compose), connect an LLM via LiteLLM, and you have a document Q&A interface in under an hour.
Dify also has a workflow mode for multi-step chains (query rewriting, retrieval, reranking, generation) that is more purpose-built for RAG than n8n's general automation. Use n8n when the output goes to another service; use Dify when the output IS the product.
The full Docker Compose setup and common migration issues are in the Dify self-hosted guide.
n8n and Dify solve different problems in a self-hosted AI stack. n8n is a workflow automation tool with 400+ prebuilt integrations and a visual AI Agent node - use it to connect AI to external services. Dify is a purpose-built RAG and chat application builder - use it when the AI application itself is the product. Both connect to Ollama or any OpenAI-compatible endpoint via LiteLLM.
Layer 5: SearXNG - private search for agents
SearXNG is a self-hosted metasearch engine that aggregates results from 262 sources without logging queries or building a profile. For AI agents that need web access, SearXNG is the private alternative to plugging agent queries directly into Google (which Google logs) or Bing.
In n8n: the HTTP Request node calls http://localhost:8080/search?q={query}&format=json and returns structured results the AI Agent node can process. In Dify: configure SearXNG as a tool in the workflow. In Agent Zero: set SearXNG as the default search provider in the config.
Setup is a single Docker command; the SearXNG guide covers the config file, engine selection, and JSON API setup for programmatic use.
Layer 6: Agent Zero - autonomous agent with OS access
Agent Zero gives an LLM full control of a terminal: it writes Python, executes shell commands, uses MCP tools, and spawns sub-agents. It's the highest-privilege layer of the stack and the one that needs the most care around sandboxing. In practical terms: use Agent Zero for jobs that benefit from iterative code execution (data analysis, file processing, multi-step research) and keep it isolated from production systems.
The exact security configuration and MCP setup are in the Agent Zero guide.
Minimum hardware to run the full stack
| Configuration | Specs | What runs |
|---|---|---|
| GPU workstation (full stack) | 16 GB RAM, consumer GPU with 8 GB+ VRAM, NVMe | Everything: local inference at 50-100 tok/s, all seven services |
| CPU workstation (no GPU) | 16 GB RAM, 8-core CPU | Full stack; Ollama inference at 3-8 tok/s, or route to a cloud API |
| VPS (cloud inference) | $40/month box: 4 vCPU, 16 GB RAM | Open WebUI, LiteLLM, n8n, Dify, SearXNG, Agent Zero; Ollama calls an external API |
| VPS (local inference) | GPU VPS: $60-120/month with T4 or similar | Full stack including Ollama local inference |
The Docker Compose approach: run each service in its own container on a shared network, with LiteLLM as the hub. Internal service names replace localhost in container-to-container URLs: Ollama is http://ollama:11434, SearXNG is http://searxng:8080, and so on.
Frequently asked questions
What tools make up a complete self-hosted AI stack in 2026?
At minimum: Ollama (local model server), Open WebUI (chat interface), and LiteLLM (API gateway). For automation: add n8n or Dify. For agent web access: add SearXNG. For autonomous code-executing agents: add Agent Zero. All seven together cover local inference, unified API management, chat, workflow automation, RAG applications, private search, and autonomous agents.
Do I need a GPU to self-host AI tools?
Only for Ollama, and even then it's optional. CPU-only inference works at roughly 3-8 tokens per second on a modern 8-core CPU versus 50-100 tokens per second on a mid-range GPU. Open WebUI, LiteLLM, n8n, Dify, SearXNG, and Agent Zero run on CPU with no performance penalty. A $40/month VPS handles the whole stack if you route inference to a cloud model via LiteLLM.
What is the difference between n8n and Dify for AI?
n8n automates workflows between tools: trigger on a Slack message, run an AI step, write to a database. It has 400+ prebuilt integrations and a LangChain-based AI Agent node. Dify builds AI applications as products: RAG knowledge bases, multi-turn chat with retrieval, document Q&A. Use n8n when the AI output feeds another service; use Dify when the chat or retrieval app is the product itself.
How do all these tools connect to each other?
Through LiteLLM and Docker networking. LiteLLM exposes a single OpenAI-compatible endpoint at port 4000; every tool (Open WebUI, n8n, Dify, Agent Zero) points its model configuration at that URL. For web search, every tool that needs it calls SearXNG's JSON API at port 8080. In Docker Compose, containers find each other by service name rather than localhost.
Can I use cloud models like Claude or GPT-4 alongside local models?
Yes, and this is one of LiteLLM's core strengths. Add your Anthropic and OpenAI API keys to LiteLLM's config alongside the Ollama endpoint. Each downstream tool can then specify a model name (e.g. claude-sonnet-4-6 or ollama/llama3.1:8b) and LiteLLM routes to the right provider. This lets you run cheap local inference for simple tasks and cloud models for complex ones from the same stack.
Written by Joshua Opolko, who self-hosts this full stack and writes about it at joshuaopolko.com. Tool versions and capabilities verified July 2026; both Ollama and n8n release frequently, so check individual guides for the latest install commands.