Local Intelligence: The Case for AI That Never Leaves Your Desk

The headline numbers in AI are all about scale: bigger clusters, bigger power contracts, bigger models. Read Flux for a week and you will see billion-dollar rounds for chips, data centres and the companies that rent them.

There is a quieter trend running in the opposite direction. A lot of useful AI no longer needs to leave the machine in front of you.

The moment it became real

In March 2023, a developer named Georgi Gerganov published llama.cpp, a plain C/C++ implementation that ran Meta's freshly released LLaMA weights on an ordinary MacBook. It was slow by data-centre standards and it was not the smartest model in the world. It did not matter. People could suddenly download a language model and run it with the Wi-Fi off.

What followed was a fast-moving ecosystem: open-weight model families from several labs, tools such as Ollama and LM Studio that turned "run a model locally" into a one-line install, and file formats designed for exactly that job. The big platforms moved too. Apple built Apple Intelligence around an on-device model of roughly 3 billion parameters, handing heavier requests to its Private Cloud Compute servers, and in 2025 opened that on-device model to app developers through its Foundation Models framework. The hardware kept pace; Flux has covered how Apple's newest silicon is being designed around these workloads.

The trick that made it fit: quantization

A model's weights are just numbers. Training usually stores them at 16 or 32 bits each. Quantization stores them with fewer bits, often four, with surprisingly modest loss in quality for many tasks.

The arithmetic is simple. A 7-billion-parameter model at 16 bits needs about 14 GB just for its weights. At 4 bits it is closer to 3.5 GB, small enough for a mid-range laptop's memory. That one technique, more than any single model release, is why "local AI" went from demo to habit.

Why people choose local

The reasons are practical, and they tend to come in the same four flavours:

• Privacy by architecture. If the data never leaves the device, there is no server log to leak, subpoena or train on. For journals, medical notes, client files and unreleased creative work, that is not a philosophical point. It is the whole point.
• Latency. No network round trip means autocomplete, transcription and small edits that feel instant, including on a plane or in a basement studio.
• Predictable cost. A model on your own machine has no per-token bill. For high-volume, low-stakes tasks, such as tagging thousands of photos or summarising your own notes, that changes what is worth automating.
• Control. You pick the model, the version and the settings. Nothing changes underneath you on a Tuesday because a provider updated something.

What local is still bad at

Honesty matters here. The biggest frontier models are far beyond what fits on a laptop, and for hard reasoning, long research tasks or the newest capabilities, the cloud is still where the best results live. Local models also cost something real: battery, heat, storage and the time it takes to keep them up to date.

The more realistic future is hybrid. Small models on the device handle the frequent, private, latency-sensitive work. Big models in the cloud handle the rare, heavy jobs, ideally with clear rules about what gets sent and why. Apple's split between on-device and Private Cloud Compute is one public version of that pattern; plenty of developer tools now route between a local model and a hosted one automatically.

A design problem as much as a technical one

The most interesting part of local AI may be what it does to product design. When the model lives on the device, designers can build features that would be uncomfortable or uneconomic in the cloud: assistants that read everything on your screen, tools that learn your personal style, creative apps that work offline in the field.

It also changes the emotional contract. A cloud assistant is a service you visit. A local one is closer to an instrument you own. That difference shows up in how people use it: more experimentation, less self-censorship, more willingness to point it at the messy, personal material that actually fills a working life.

The small machine with the big room inside

There is something quietly poetic about all this. The same industry that is pouring concrete for gigawatt campuses is also proving that a surprising amount of intelligence fits in a backpack. A modest computer on a desk, lights off, holding a model that can read, write, sort and sketch for you without telling anyone.

That will not replace the frontier. It does not need to. For a lot of everyday work, the most useful AI may simply be the one that never has to leave the room.

---

A modest computer holding a world larger than its frame is the idea behind our Local Intelligence tee at AERIOX.