The Cheapest AI That Actually Works

Most advice about running your own AI model is written by people selling hardware. We wanted to know something simpler: what's the cheapest thing that actually works for a small business website?

So we tested it on ourselves. We ran Gemma locally while building three features: an ask tool for our site, an audit tool, and an enrichment pipeline. Local was the right place to do that, because we could experiment without watching a meter.

What we learned is where the line is.

The line turned out to be uptime

We run more than one model, and only one card can hold a decent one at a time, so every model we loaded evicted another. Our ask feature checks whether Gemma is actually available and hides itself when it isn't. That's an honest answer for a prototype and a bad one for a service your customers are using.

So we moved the live features to OpenRouter. Across our enrichment pipeline, our code tooling and the ask feature, the cost has stayed genuinely nominal.

Swapping a model is not a drop-in

The obvious fix for eviction is to put a second model on a second card, so we tried it. Qwen at 14 billion parameters went on the spare GPU to cover the ask feature while Gemma handled other work.

It did not hold up. Asked questions about our own articles, it drifted from what the source material actually said, confidently, in a way that would have been embarrassing in front of someone genuinely trying to find something out. There is a difference between a model getting something slightly wrong and a model inventing an answer, and only one of those is survivable on a public feature.

This was not a misuse of the model. It is a well regarded one, and answering questions over a supplied set of documents is a standard thing to ask of it. What we had not appreciated is that the shape of the question matters more than the reputation of the model. Pulling a single fact out of a single article is a far easier job than reading across several articles and combining them into one answer, and ours does the second. A model can be perfectly good at the first and unreliable at the second, and nothing on the label tells you which one you are about to rely on.

So the practical advice is duller than picking the right model: write your test suite first. Twenty or thirty real questions with the answers you would want, run them against any model you are considering, and read the output yourself. It takes an afternoon and it is the only thing that tells you whether a model can do your job rather than a job.

Our bar was not a benchmark score. It was whether we would be comfortable putting the answer in front of a customer. Gemma cleared that. Qwen, on this particular task, did not.

Three answers, not one

So we ended up with three answers rather than one. Run a model locally when you're figuring out what you need, or when the behaviour matters enough that you want it pinned: a vendor upgrade can quietly undo months of tuning. Use a cheap open model through OpenRouter when the job is bounded, meaning a fixed set of documents, a known shape of question, unstructured data going in and structured data coming out.

Pay frontier prices when the work is open-ended.

That last one is worth being clear about, because it is where most of the real money goes and it is the easiest to talk yourself out of. Open-ended work is not just a harder prompt. It comes with machinery around it: a loop that can call tools, hold a growing context, and correct itself when a step goes wrong. That is the difference between a model that answers a question and one that does a job, and it is why our own code assistance runs on a capable model rather than the cheapest tokens we can find. The cost of moving away from it is not really the subscription. It is that a cheaper setup changes how you work, and relearning that is time not spent on the actual job.

What this means for you

None of this requires you to own a graphics card or read a benchmark. It requires one decision, made per task rather than once for everything: is this job bounded, or is it open-ended? Get that right and the bill for the bounded majority stays close to nothing, which leaves the budget for the few places where the expensive version actually earns it.