Local-first AI: for teams that cannot send data to APIs

Some data cannot go to an API. Client contracts, NDAs, internal files that legal would rather never leave the building. The vendor answer is an enterprise tier, which still means the bytes leave your machine. Builder-land has a working answer now, and it stopped being a tinkerer’s diary this month. On September 1, a complete local AI stack on a Mac mini hit Hacker News’s front page. It pulled 335 points and 199 comments (HN), because every layer of the stack is commodity in 2026.

The stack that made Hacker News argue

Kevin Lewis runs the reference setup. His machine is an M4 Pro Mac mini with 48GB of unified memory (his write-up). An oMLX server listens on port 8000, and a Tailscale tailnet connects his phone, laptop, and agent backend. Nothing touches the public internet. He serves two models. Qwen3.6-35B-A3B in 4-bit quantization takes about 20GB of RAM, while a 2.4GB Gemma-4-E4B handles routine chat. His measured numbers are 325 tok/s on prompt processing and 34 tok/s on generation. He calls that quick enough that he never thinks about it, which is the right bar for daily use.

His four stated reasons read like the privacy buyer’s ledger. Cloud APIs are rented land, because pricing, limits, and the model behind the endpoint can change without notice. Data privacy comes second. His key line: sending client data to a third-party API is “a decision you make once and cannot undo”. Sovereignty and cost predictability fill out his list. I run my own tools on a self-hosted VPS, because flat cost and control matter to me. I don’t handle client contracts, yet the shape is identical: own the loop, rent the frontier.

The four layers, and what each one does

Every working local-first setup reduces to the same four components. You are choosing brands, not inventing architecture, which is why the boring parts are already solved. Ollama alone sits at 180,454 stars, and llama.cpp at 127,480 (GitHub, llama.cpp). Both figures are as of September 8, 2026. Both speak the OpenAI-compatible API format, which means any hosted-model client just repoints at a local URL. Swap the box out and every client keeps working, because the interface never changes.

The RAM math that decides the purchase

One number decides whether a machine can serve a model class. That number is unified memory, because bandwidth only sets how pleasant the experience feels. The anchor post does the arithmetic cleanly. Look at the 4-bit file size, which runs roughly at the parameter count in gigabytes. Subtract the 6-8GB that macOS takes. Add 8-16GB of headroom for the KV cache when you expect long sessions. Anything closer than a 10-15% buffer to full will swap to SSD and get painful.

The M4 Pro’s 273 GB/s bandwidth sets the speed limit, so the box feels usable rather than instant. Prompt processing is the other tax, because prefill runs slower than generation on Apple Silicon. One M3 Ultra owner calls it the main issue for M1-M4 machines (HN).

Mixture-of-experts models are the reason consumer boxes work at all. A dense 27B loads every parameter for each token. The 35B-A3B spreads 35 billion across experts while activating about 3 billion per token. Its real footprint behaves like a much smaller dense model, which is the whole ballgame on a 16GB MacBook Air. A dense 27B at 4-bit needs roughly 14GB there, so the OS gets nothing and the swap kills it. The failure mode is not hypothetical. One commenter describes a Fortune 20 lead engineer who bought a Mac for a 70B model in 2025 (HN). It failed, and an A6000 proposal eventually worked. Buy memory for the model class you will actually serve.

• Check the quantized file size first: 4-bit runs at roughly the parameter count in GB.
• Subtract 6-8GB of macOS overhead from the machine’s total memory.
• Add 8-16GB for the KV cache when you expect long agent sessions.
• Prefer MoE models under 64GB, since active parameters set the real footprint.
• Keep a 10-15% memory buffer, or the box will swap to SSD under load.

What it is actually good for

The people running these boxes converge on the same verdict. Local covers the routine requests, and the frontier stays rented. Lewis states it directly: the point is the “80% of requests that do not need GPT-5 or Claude Opus”. Speed reports back the same story. One M4 Pro user measures 52 tok/s on a Gemma-class model, and 72 on a Qwen-class. Both run 4-bit, and he calls them good enough, though he wouldn’t use them for coding features (HN comment). His use list covers offline reference, reformatting, translation work, and learning new topics, which is the routine band. Multiple users independently report killing the models’ thinking mode to get usable speed. The reasoning tokens burn the clock. Turn it off unless the task really needs the depth, because the waits get long.

The marketing translation is the part nobody else writes down. Some tasks move local cleanly: first-pass copy drafts, summaries of internal documents, and format conversions. Call transcription belongs on this list when an NDA covers the call. Q&A over private playbooks works the same way. Tasks that stay on the API are the ones that need live search, campaign-scale agent loops, or frontier reasoning. The review layer travels with you either way, because a local draft still needs a gate. Before anything reaches a client, run it past the same human-in-the-loop gate I described earlier.

The two edge cases: heretic and WebLLM

The local world’s loudest repository removes safety training from models. That fact has a shadow side which the team-facing coverage rarely connects. Heretic, the tool doing it, sits at 30,888 stars as of September 8, 2026 (GitHub). Its method is abliteration: directional ablation steered by an Optuna parameter search. The README’s table shows gemma-3-12b-it dropping from 97/100 refusals to 3/100 on harmful prompts. It runs from one CLI command, per the docs, and it already supports several MoE architectures. One HN user ties it to the moment, citing ideology fixation in both the US and China (HN).

WebLLM is the other edge case, and its September 2 thread turned on it fast (HN, 145 points). Long-time users report the project stalled. The top comment says “Project is de facto dead” after years of use. Another commenter pins the stall to NVIDIA’s acquisition of OctoAI, which closed in late 2024. The repo still shows 19,025 stars and recent commits as of September 8, 2026. Treat the abandonment claims as individual reports, since I cannot verify the maintenance state myself. Browser inference still has one honest niche, which is zero-install demos. One practical note stands: a session downloads 500MB to 1GB of weights.

The decision rule

One rule decides this, and it has two clauses. If the data cannot leave the building, the box pays for itself in reduced risk. The 80/20 split is the plan in that case, not a compromise. If frontier performance is the requirement, an API stays the better buy at nearly any usage level. The recent Fable 5.1 pricing read shows how cheap repeated-context work has gotten on the hosted side. Most teams will run both, exactly as the anchor post does. Keep the sensitive loop on your own infrastructure, which is the same instinct behind self-hosted n8n marketing agents. Rent the frontier when the task is worth it, and read the two-vendor API stack for the hosted side. Cloud is still the right frame for most work, as what ChatGPT is actually good for argues.