Building a machine for local AI

This is not a gaming build with a language model bolted on. The machine described here has one job: hold a model in memory and answer requests all day without sounding like a hairdryer. That changes which parts matter and which ones you can stop caring about.

Before the parts list, the rule that decides everything: memory on the accelerator sets what you can run; memory bandwidth sets how fast it feels. Everything else is support.

Three shapes, three budgets

Mini PC with shared memory. One box, low power, quiet. Runs small models, embeddings, speech-to-text and the agent runtime itself. Nothing to assemble, nothing to upgrade except RAM and storage. If your models are in the 7–8B class, this is where most small companies should start: Beelink SER8 on Amazon.

Desktop with 16 GB of dedicated VRAM. The build this page is about. A 16 GB card holds mid-size quantised models with room for a real context window, and the whole machine stays a desktop: one PSU, one warranty, sits under a desk.

Workstation with 24 GB or more. Only when you have measured that you need a 30B-class model at useful context, or several small models loaded at once. Above 24 GB the price curve turns steep and the server form factor brings noise and heat into your office.

What each part actually does

GPU — the only part that decides capability. VRAM first: 16 GB is the first honest step, 24 GB is comfortable for 30B-class models at 4-bit. Second, memory bandwidth, which is what generation speed tracks. Third, cooling: a triple-fan card at 300 W is audible in a quiet room, so pick the case and the fan curve with that in mind.

The card we used in the last build: PNY GeForce RTX 5070 Ti OC — 16 GB GDDR7 on a 256-bit bus, PCIe 5.0. Note what matters in that sentence: 16 GB and a wide bus. Ray tracing and upscaling, the things reviews lead with, are irrelevant here.

CPU — support, not capability. It feeds the GPU, runs the OS, the containers and the agent runtime. For local inference you want enough cores and a sane power envelope, not the fastest gaming chip. A 3D V-Cache part like the Ryzen 9 9900X3D is excellent hardware and we use it, but be honest with yourself: its premium buys game frame rates, not tokens per second. A cheaper non-X3D chip at the same core count would leave the model performance essentially unchanged.

RAM — 2×32 GB desktop memory. Not for the model, which lives in VRAM, but for the OS, containers, document pipelines and the CPU-offloaded layers that appear when a model does not fit. Two matched sticks, in the correct slots, with XMP/EXPO enabled. Do not flash money on extremely high clocks: capacity matters here, speed barely.

Storage — the models are the bulk. A 2 TB NVMe drive holds a handful of models plus the system with room to spare. The Samsung 990 PRO 2 TB is a known-good choice: 7,450 MB/s sequential, PCIe 4.0, M.2 2280. Sequential speed only matters when loading a model; once it is in VRAM it does not touch the disk. Wear is a non-issue at this workload, so a cheaper drive is perfectly defensible.

PSU — headroom for spikes. Modern cards draw brief peaks far above their nominal power; a weak supply turns those into crashes that look like software bugs. For a 16 GB card, 850 W from a reputable brand, ATX 3.x with a native 16-pin cable, is the sane bracket. Do not reuse a ten-year-old unit to save on the one component that can take the rest of the machine with it.

Case and cooling. You will hear this machine every day. Two 140 mm intakes, one exhaust, dust filters, and a case that fits the card length you are buying. Check the clearance before ordering, not after.

Motherboard. One PCIe x16 slot wired with full lanes, two M.2 slots (one for models, one for the system), and enough USB for the UPS signalling cable. Chipset features you will not use — extra PCIe lanes for a second card you are not buying, eight SATA ports — cost money and buy nothing here.

Assembly order

1. Build outside the case first: motherboard, CPU, one stick of RAM, cooler. It either posts or it does not, and finding out is much easier on a box than in it.
2. Update the BIOS before you install anything else. New CPUs and memory kits frequently need it.
3. Install the CPU cooler with the right mounting kit, then the second stick of RAM in the paired slot.
4. Card in the top slot, PSU in, cables routed behind the tray, native 16-pin for the GPU.
5. First boot into the BIOS: confirm the card is detected, memory at full capacity and speed, fans behaving.
6. Install the OS on the system drive. Keep models on the second drive — a separate mount, so a reinstall does not mean re-downloading weights.
7. Install the GPU driver, then the runtime, then pull the first small model. Test with a real question, not “hello”.
8. Run a sustained load for twenty minutes and watch temperatures, clocks and fan noise. If it throttles now, it will throttle in production.

The measurements that decide whether the build worked

• Tokens per second at the context length you actually use. Not the empty-context number from a benchmark.
• VRAM headroom while serving: if you are at 95 % with one user, the second user gets an error or a much slower path.
• Power draw at idle and under load — the difference tells you what the box costs per month, and whether the PSU has headroom.
• Temperature and noise under sustained load, measured in the room where the machine will live.
• Time to first token with a full context window, because that is what people perceive as “slow”.

Write the numbers down. They are the baseline you will compare against after the first model update.

If the budget is limited, buy in this order

1. Storage and RAM on the machine you already have. Cheapest useful step, and it makes CPU inference with small models possible.
2. A 16 GB GPU. The single biggest capability jump for the money.
3. Platform (CPU, motherboard, PSU), if the old one cannot feed the card.
4. A UPS. Ranked last by most people and first by anyone who has lost a long job to a power cut: APC Back-UPS Pro 1500VA on Amazon.

What we deliberately did not put on this page

No prices, because they move weekly and an outdated number is worse than none — check them on Amazon. No benchmark tables of models, because the model landscape changes faster than a page like this stays true, and because your test set matters more than a leaderboard.

And no claim that this build is right for everyone. If you cannot describe the workload in tokens per day and model size, the correct next step is a smaller experiment, not a bigger machine. Which is exactly the conversation we have before quoting any AI agent project: tell us the four numbers, and we will say whether to build, rent, or wait.

The software side — which runtime, how quantisation works, how to serve an API your agents can call — is on how to run your own local AI.