Every second client conversation starts the same way: “can we run this ourselves instead of paying per token?” Sometimes yes, and the deciding factor is rarely the CPU. It is memory — how much of it the accelerator can address, and how fast the whole box can feed it. This page is the arithmetic we go through before recommending anything, followed by the parts that survive that arithmetic for a small business.
If you want the short version: a compact desktop with shared memory runs small models and agent plumbing; a machine with 16 GB of dedicated VRAM is the first honest step for larger local models; anything above that stops being a purchase decision and becomes a capacity plan.
When local makes sense, and when it does not
Local inference wins when the workload is steady, when data cannot legally leave your premises, or when latency and availability matter more than the marginal cost per request. It loses when usage is spiky: a box that idles nine hours a day still costs its purchase price, while an API bill for the same work may be a few dollars a month.
The uncomfortable middle is the most common one. Companies buy a machine “for AI”, run a handful of agent jobs on it, and then pay twice — once for hardware, once for the cloud calls that still handle the heavy work. Before buying anything, measure four numbers for a month: daily token volume, peak concurrent requests, the largest model you actually need, and how much of the data is genuinely sensitive. If the last number is zero, cloud is usually cheaper.
The only arithmetic that matters
Model weights must fit in memory. A useful planning rule for quantized weights is roughly 0.5–0.6 bytes per parameter at 4-bit quantization, plus room for the KV cache, which grows with context length. That gives:
| Model size | Quantized weights | Realistic machine |
|---|---|---|
| 7–8B | ~4–5 GB | any mini PC with 16 GB of shared memory |
| 13–14B | ~8–9 GB | mini PC with 32 GB, or a 12 GB GPU |
| 30–34B | ~17–19 GB | 16 GB GPU does not fit it comfortably; 24 GB does |
| 70B | ~40 GB | workstation-class memory, or several GPUs |
Two consequences people miss. First, shared memory on a mini PC is not free: the GPU partition is carved out of system RAM, so a 32 GB machine with 8 GB allocated to graphics leaves 24 GB for everything else, including your containers. Second, VRAM is not only about fitting the weights — the context window you promise your users lives in the same budget.
Three shapes of machine
A compact desktop. One box, low power, quiet, no discrete GPU. It runs small models, embeddings, speech-to-text, schedulers and the agent runtime itself. This is the right first purchase for most small companies, because it also replaces an always-on laptop that someone was using as a server.
A desktop with 16 GB of dedicated VRAM. The first configuration where a mid-size quantized model runs at a speed people tolerate. It stays a desktop: one power supply, one warranty, no rack. If your work is document processing, transcription, local assistants, or an agent that calls tools and reads files, this is usually the point of diminishing returns.
A workstation or a server. Only after the arithmetic says you need 40 GB or more, and only with a concrete capacity estimate. Buying a server “to be ready” is how companies end up with a noisy box in a closet and an unchanged cloud bill.
What we would buy today
The three items below are the ones that fit the reasoning above without pretending to be a benchmark. We list what they are for and what they will not do. Prices change constantly, so check them on Amazon rather than trusting a number on any review site, including this one.
The part nobody budgets for
Power. A machine that runs long jobs and gets interrupted mid-run can corrupt a fine-tuning run, a database, or an agent’s working state, and a UPS costs a fraction of the time lost. For a single-box setup, a 1500 VA tower unit with automatic voltage regulation is the sane default: APC Back-UPS Pro 1500VA on Amazon.
What this means for your project
Buying the hardware is the small half of the job. The rest is the runtime: which model, which quantisation, how you route requests between local and cloud, how you keep prompts and outputs out of logs you do not control, and how you monitor a box that has no support contract. That is what we build for clients — AI agents and MCP servers with the infrastructure decision made explicitly, not by accident.
If you are weighing local against cloud, tell us the four numbers — volume, concurrency, model size, sensitivity — and we will tell you which side of the line you are on, including when the answer is “stay in the cloud”.