Running a model on your own machine stopped being an experiment and became an ordinary infrastructure decision. What follows is the path we take with clients who want their agents, their documents and their prompts on metal they control — in the order the decisions actually matter, with the failure modes we keep meeting in real installs.
If you have not bought anything yet, start with the hardware arithmetic instead: choosing software before knowing how much memory you have is how people end up with a model that does not fit.
What “local” really means
Local means two things at once: the model weights are files on your disk, and every token is produced by your own processor. Nothing about the prompt, the retrieved documents or the answer leaves the machine unless you send it. That is the whole commercial argument when data is sensitive, and it is also the reason local setups fail in boring ways — disks fill, fans spin, machines reboot.
Local does not mean free. You pay once for hardware, continuously for electricity, and with engineering time for the runtime, updates and monitoring. An honest comparison against an API bill has to include all three.
Four decisions, in this order
- Model and size. What has to be done: general chat, document extraction, classification, code, embeddings, speech? Each has small specialised models that beat a big general one on the task, at a fraction of the memory.
- Quantisation. How many bits per weight you accept. This sets the memory floor.
- Runtime. The program that loads the weights and produces tokens.
- Serving. How other software — your app, your agent, your colleagues — reaches it.
Doing them in another order (buying a runtime first, or a GPU first) is the usual source of waste.
Choosing the model
The practical filter is not “which model is best”, it is “which model is good enough at this one job, and fits”. Three categories cover most small-business work:
Small general models (7–14B parameters). Document summarisation, drafting, routing, tool calling, classification with a bit of reasoning. These run on ordinary hardware, including laptops, which is why they are the right first experiment.
Mid-size models (30B class). Noticeably better at multi-step instructions and longer context, and the first size where a dedicated GPU with 24 GB of memory starts to matter.
Specialised small models. Embeddings for search, speech-to-text for calls, tiny classifiers for routing. In practice these do most of the useful work in an agent pipeline, and they cost almost nothing in memory compared with a chat model.
Do not start with the largest model you can theoretically run. Start with the smallest that passes your own test set — twenty real documents or twenty real questions from your business, with the answers you consider correct. That test set is worth more than any leaderboard.
Quantisation: the number that decides everything
Weights are stored with a certain number of bits each. Full precision (16-bit) costs 2 bytes per parameter; 8-bit costs 1; the 4-bit formats people actually use cost roughly 0.5–0.6 bytes per parameter. So a 7–8B model lands near 4–5 GB, a 30B model near 17–19 GB, a 70B model near 40 GB.
Two things sit on top of that number:
- KV cache — the working memory for the conversation context. It grows with context length and with the number of simultaneous requests, and on long-context workloads it can rival the weights.
- Runtime overhead — buffers, the compute graph, the framework itself.
The quality difference between 4-bit and 8-bit exists but is task-dependent; measure it on your test set rather than trusting a rule. What you should not do is run a bigger model at 3-bit because it barely fits — degradation there is usually worse than choosing a smaller model at 4-bit.
Picking a runtime
Four options cover almost every case:
| Runtime | What it is good at | What to watch |
|---|---|---|
| llama.cpp / GGUF | Runs on CPU, Apple Silicon and GPU; the widest hardware support; fine control over memory | Less convenient to package; you own the serving layer |
| Ollama | The fastest way to a working endpoint: one binary, model pulls by name, OpenAI-compatible API | Defaults favour convenience over throughput; pin versions in production |
| LM Studio | Desktop experiment with a GUI, model browser and a local API | Not a server; fine for evaluation, not for a service |
| vLLM | High throughput on GPUs, batching, production serving | Needs a real GPU and more setup; overkill for one user |
The usual progression is honest: LM Studio or Ollama while you are evaluating, then Ollama or llama.cpp on the machine you keep, and vLLM only if several people or agents will hit the same model at once.
The first run, step by step
- Install one runtime. On a fresh machine, Ollama is the shortest route to something that answers.
- Pull a small model first — a 7–8B quantised build is enough to prove the pipeline.
- Ask it something from your test set. Not “hello”: a real question with a known answer.
- Watch memory while it answers. If the model is partly on the CPU, generation gets dramatically slower; that is the signal your quantisation or model size is wrong for this machine.
- Note the speed in tokens per second at your usual context length. This number, not a benchmark table, tells you whether people will tolerate the setup.
- Only then try the bigger model, and repeat.
Serving it to the rest of your software
Every runtime above can expose an HTTP endpoint, and most now speak the OpenAI-compatible chat-completions shape. That matters because it lets you move between a local model and a hosted one by changing a base URL, not by rewriting the application.
What to put in front of it:
- A reverse proxy with TLS — never expose a bare model port to the internet.
- Authentication — one token per consumer, rotated; a local model endpoint with no auth is an open door on your network.
- Rate limiting and a request timeout — one runaway agent loop should not occupy the machine for an hour.
- Logging that you thought about — prompts and outputs often contain personal data; keep logs short-lived and outside the model server.
- A queue or concurrency limit — a single model instance serves one request at a time efficiently; parallel requests either queue or slow each other.
Connecting it to an agent
A local model becomes useful when something calls it as a tool-using component: an agent runtime that reads files, queries your database, or drives a workflow. Two practical notes from our own builds:
- Tool calling is the weak spot of small models. Test it explicitly. If a 7B model calls the wrong tool under pressure, either move to a larger model or keep the tool surface small and validate arguments before execution.
- This is where MCP fits. An MCP server gives the model a typed, auditable way to reach your systems, instead of a pile of ad-hoc API calls inside prompts.
Operating a box nobody supports
A local model server has no vendor, no status page and no support contract. The minimum we insist on:
- Monitoring: is the endpoint answering, how long are requests taking, how much memory is free, how hot is the GPU.
- Restart policy: the service should come back after a reboot without a human.
- Disk discipline: models are large files; two or three versions plus logs will surprise you.
- Power: a machine that runs long jobs and loses power mid-run can corrupt state. A UPS is cheaper than the rerun — see the hardware page for the one we use.
- A written rollback: keep the previous model file and the previous config, so an update that degrades quality is reversible in minutes.
The failures we see most
| Symptom | Usual cause |
|---|---|
| Generation suddenly slow | Model partially offloaded to CPU, or context grew past the fast path |
| “Out of memory” on a bigger model | Quantisation too aggressive for the KV cache you also need |
| Good answers in chat, bad in the app | Prompt and tool descriptions differ from what you tested |
| Works for one user, collapses for three | No concurrency limit; one instance, several callers |
| Nobody uses it after a month | No integration into the actual workflow — a chat window is not a product |
When the answer is “stay in the cloud”
If usage is spiky, if the model you need exceeds the memory you can justify buying, or if nobody on the team wants to own a server, cloud inference is the correct answer and local is a hobby project with a rack. We say this to clients regularly, and it saves them money.
Our own rule: local first where data cannot leave or latency must be predictable; cloud first where the workload is bursty. Mixed setups work well — small local models for embeddings, routing and extraction, a hosted model for the heavy reasoning.
If you want an opinion on your specific case, send us the four numbers: daily volume, peak concurrency, model size, and how much of the data is genuinely sensitive.