AI That Stays on Your Server and Knows Your Business

Most conversations about adding AI to a business start in the same place: which API should we use? OpenAI, Anthropic, Google, the assumption is that AI means paying a third party per token, sending your data to their servers, and trusting their privacy policies to hold up under scrutiny.

That's one path. There's another one where no API key is needed at all.

What we built and why

We recently added an AI assistant to our own site at bluemonkeymakes.com/ask. It answers questions about our work, our services, case studies, articles, demos, grounded in our published content. It declines off-topic questions. And nothing leaves our servers.

The stack:

• Ollama, an open-source LLM runtime that runs models locally on standard hardware
• Open WebUI, an orchestration layer that handles the knowledge base, retrieval, and model management
• Gemma 3 (27B parameters), an open-source model from Google, running entirely on our infrastructure
• Custom chat UI, built into our Remix app, not an iframe or third-party widget

The whole thing runs on our existing Hetzner infrastructure. The same servers we were already paying for. The incremental cost in API fees is zero.

We built it partly because we wanted to understand how self-hosted AI actually works in practice, and partly because we kept seeing the same disconnect in how businesses think about AI: the conversation always centers on which vendor to pay, never on whether a vendor is needed at all.

The default path is expensive and leaky

The standard approach to adding AI capabilities looks something like this:

• Sign up for an API key with a major provider
• Pay per token, input tokens and output tokens, priced separately, adding up fast with any real usage
• Send your business data to the provider's servers with every request
• Hope the vendor's data retention and training policies align with what you told your customers about privacy
• Accept that your costs scale linearly with usage, with no ceiling

For many use cases, this is perfectly reasonable. Large language models from the major providers are remarkably capable, and for complex reasoning tasks they remain the best option available.

But for scoped business applications, answering questions about your own content, searching your own knowledge base, handling customer support grounded in your own documentation, the cloud API path introduces costs and risks that aren't always justified by the capabilities you actually need.

Open-source models have caught up for scoped tasks

The gap between proprietary and open-source models has narrowed dramatically over the past year. Models in the 7B to 27B parameter range, small enough to run on a single server, are genuinely useful for focused tasks.

They are not as capable as the largest cloud models for open-ended reasoning, creative writing, or complex multi-step analysis. That distinction matters. But for retrieval-augmented generation, content search, FAQ answering, and knowledge base queries, they perform well. Often well enough that the difference is invisible to the end user.

The model we run, Gemma 3 at 27 billion parameters, handles our use case cleanly. It understands questions, retrieves relevant context from our knowledge base, and generates coherent, grounded answers. It occasionally lacks the polish of a frontier model, but for a scoped assistant that answers questions about our own business, polish is less important than accuracy and privacy.

RAG: making the model know your content without training it

One of the most common misconceptions about AI is that you need to "train" a model on your data. For most business applications, you don't. What you need is RAG, Retrieval Augmented Generation.

Here's how it works in plain terms:

• Your content (articles, documentation, service descriptions, case studies) gets loaded into a knowledge base and broken into chunks
• Each chunk gets converted into a numerical representation (an embedding) that captures its meaning
• When someone asks a question, the system converts that question into an embedding too, then searches for the most relevant chunks
• The relevant chunks get passed to the model as context alongside the question
• The model generates an answer based on that context, not from memory, but from the actual content it was just given

The model doesn't memorize your content. It doesn't need to. It searches it at query time and uses the relevant pieces as context for its response. This means updating the knowledge base is as simple as adding or editing documents. No retraining. No fine-tuning. Change the content, and the next query reflects it.

Scoping: the AI should only know what you want it to know

An AI assistant for a business should answer questions about that business. It should not write Python scripts, give medical advice, or debate philosophy. This sounds obvious, but it requires deliberate scoping.

We handle this through system prompts with explicit rules:

• The assistant answers questions about Blue Monkey Makes, our services, articles, case studies, and demos
• It draws only from the loaded knowledge base, not from its general training data
• It declines off-topic requests with a clear, polite explanation
• It does not fabricate information when the knowledge base doesn't contain a relevant answer

That last point matters more than people realize. A general-purpose model will cheerfully make up plausible-sounding answers when it doesn't know something. A scoped assistant should say "I don't have information about that" instead. The graceful decline is a feature, not a limitation.

Where self-hosted AI fits for small businesses

The pattern we built, an open-source model, running locally, connected to a curated knowledge base, is more versatile than a single chat widget:

• Customer-facing content assistant. What we built. Visitors can ask questions about your work and get answers grounded in your actual content. It's like a search bar that understands natural language and gives synthesized answers instead of a list of links.
• Internal knowledge base search. A team can ask questions about SOPs, processes, documentation, and institutional knowledge in natural language. Especially valuable when information is scattered across multiple documents and nobody remembers where anything lives.
• Sales qualification. A prospect-facing assistant that can answer "Is this business a good fit for what I need?" based on your published services, case studies, and ideal client descriptions. Not a replacement for human conversation, but a useful first filter.
• Website content exploration. For businesses with a deep content library, articles, case studies, product descriptions, an AI assistant lets visitors navigate by intent rather than by navigation structure.

The common thread is that these are all bounded tasks with a defined knowledge base. The model doesn't need to know everything. It needs to know your content and stay within it.

Where it doesn't make sense yet

Self-hosted AI is not a universal solution:

• Tasks requiring real-time data. A local model connected to a static knowledge base cannot answer questions about today's stock price, current weather, or live inventory levels without additional integration work.
• Complex multi-step reasoning. For tasks that require long chains of logic, synthesis across many domains, or nuanced judgment, frontier cloud models still have a meaningful edge.
• High-consequence decisions. Anywhere an error has significant cost, medical, legal, financial, the current generation of open-source models does not provide the reliability needed.
• Very high throughput. Cloud APIs are optimized for massive concurrent usage. A single server running a 27B model handles requests sequentially. For hundreds of simultaneous users, the cloud scales more gracefully.

Knowing where the tool fits and where it doesn't is more useful than pretending it fits everywhere.

What self-hosted AI actually costs in practice

Running your own AI costs in practice:

• Speed. A self-hosted 27B model on a VPS is slower than a cloud API call. Response times are measured in seconds, not milliseconds. For conversational use cases this is acceptable. For real-time applications it may not be.
• Hardware. Running a 27B parameter model requires a server with 32GB or more of RAM. Smaller models (7B-13B) run on less, but with reduced capability.
• Infrastructure knowledge. Someone on the team needs to be comfortable with Docker, Linux, and the general shape of how these tools fit together.
• Maintenance. Models get updated. Tools get updated. The knowledge base needs refreshing when content changes. None of this is hard, but it's ongoing.

For a technical team that already manages its own infrastructure, which describes us, these trade-offs are manageable. For a team that doesn't want to think about servers, the managed API path is probably the right choice.

Privacy as a feature

When every query and every response stays on your own servers, privacy stops being a policy document and becomes an architectural fact. There's no data processing agreement to read. No question about whether a vendor is using your data to train future models.

For client-facing work, this matters. When we tell a client "your data never leaves our servers," that's not a marketing claim. It's a description of how the system is built. The model runs locally. The knowledge base is local. The conversation history is local. There is no third party in the loop.

The value is in the integration

The model itself is the least interesting part of what we built. Gemma 3 is freely available. Ollama takes minutes to install. Open WebUI is a single Docker container.

What makes the system useful is the integration: the knowledge base curated from our actual content, the scoping rules that keep the assistant focused, the custom UI that fits naturally into our site, and the infrastructure that keeps everything on our own servers.

An AI that knows your business and stays on your infrastructure is more useful, for most practical purposes, than a general-purpose model that knows everything and owns nothing. The frontier models are more capable in the abstract. But capability without context is just a party trick. An assistant that reliably answers questions about your specific work, grounded in your specific content, running on your specific servers. That's a tool.

The technology is open source, the hardware requirements are modest, and the ongoing cost is close to zero. The investment is in the integration work: deciding what the assistant should know, loading that knowledge, defining the boundaries, and building the interface. That's where the value lives.