Building a Local AI Boilerplate That Stays on One Machine
How we took a Docker-based local AI stack (Ollama, SearXNG, Open WebUI) from a working draft to a hardened, single-device boilerplate, step by step.
We set out to build a boilerplate for an AI assistant that lives entirely on one machine: no accounts, no API keys, no cloud service required. This post covers what the stack is, every change we made to harden it, and where the project stands now that it has moved into testing.
What the stack is
Three services run together under Docker Compose. Ollama runs the language model (we used llama3.1). Open WebUI is the chat interface. SearXNG is a self-hosted meta-search engine that gives the assistant optional access to live web results. Everything is bound to localhost, so nothing is exposed to your network by default.
A setup script handles first-run work, and a system prompt file defines how the assistant behaves.
How local is local?
After the one-time downloads of the container images and the model, the core stack is designed to need nothing from the outside. The one exception is SearXNG, which is optional. When you use it, it queries public search engines on your behalf, so those searches do leave the machine. Leave it out and the assistant runs fully self-contained.
Get the project
The current version of the boilerplate is available as a single zip. It matches the steps described below and is still in testing.
Download sovereign-ai-boilerplate.zip
Before you run it, read through the compose file and the system prompt so you know exactly what it does on your machine.
The steps we took
- Defined the goal. One device or server, fully self-contained, with no accounts or outside services needed for normal use.
- Audited the original boilerplate. We reviewed the compose file, the SearXNG settings, the system prompt, and the setup script under one rule: upgrades and efficiency fixes only, with no loss of functionality.
- Added health checks. Open WebUI now waits until Ollama and SearXNG report healthy, instead of starting as soon as their containers exist.
- Turned telemetry off. The stack sets ANONYMIZED_TELEMETRY=False and DO_NOT_TRACK=1 so nothing phones home by default.
- Hardened all three containers. Ollama and Open WebUI now drop all Linux capabilities and enable no-new-privileges, matching what SearXNG already had.
- Removed the obsolete version key from the compose file, which modern Docker Compose ignores with a warning.
- Made the setup script safer. It now stops on errors, checks that Docker is installed, avoids overwriting an existing SearXNG key when re-run, and times out instead of waiting forever for Ollama to start.
- Added SearXNG timeouts and retries so one slow search engine cannot stall a query indefinitely.
- Kept image tags on latest, with a marked line for pinning. That is fine for a first run. Pin exact versions when you want reproducible builds.
- Added a session secret, then took it back out. We first generated a WEBUI_SECRET_KEY in an .env file. It was only a locally generated string for signing session cookies, but it was unnecessary: Open WebUI creates its own key and stores it in its persistent data folder. We reverted, so there is no .env file and no manual key handling.
- Moved the project into a real folder in VS Code. We set up and added the Docker and YAML extensions for container status and config validation.
- Added a .gitignore. It keeps the data folder (chat history, uploads, local accounts) and any .env files out of version control.
- Ran setup and checked each service on its own. Before any end-to-end use, we ran to confirm all three containers were healthy and the model was installed.
- Removed the remaining third-party dependencies. After the IDE pass, the only outside service the stack can use is the optional SearXNG.
- Gave the assistant an onboarding routine. The system prompt now has it begin by asking the user to give it a name, then asking what it is supposed to learn.
- Moved to testing. The next test is a question that needs a live web search, run with the container logs open so the full path from SearXNG to Open WebUI to Ollama can be watched.
Where it stands
The boilerplate is in testing, so treat the claims above as design intent until the results are in. We will report what the logs show.
Two housekeeping notes for anyone building from this: keep the data folder out of git, since it holds your chat history, and consider pinning image versions once you have a configuration you trust.