The most useful shift in practical AI tooling this year is not a smarter model. It is the growing acceptance that you do not have to choose between running everything locally and sending everything to a cloud API. Hybrid compute — local by default, cloud on escalation — is quietly becoming the standard architecture, and as of September 2026 it is available in shipping products rather than research papers.
This guide covers how to build that split yourself, what hardware you actually need, and where the approach still falls short.
Start With the Local Stack
The tooling has stabilised into a small number of reliable options, which is a mercy compared with two years ago.
Ollama is the sensible default. It installs on macOS, Linux, Windows or Docker, pulls a model in a single command, and exposes an OpenAI-compatible API on localhost:11434 — meaning any application that already speaks the OpenAI dialect can be pointed at your machine with a one-line configuration change. That compatibility is the single most important property of the local stack, because it means you are not rewriting integrations to experiment.
LM Studio is the GUI-first alternative, free for both personal and commercial use, and it also exposes an OpenAI-compatible local server once you enable developer mode. If you would rather browse models than read model cards, start here.
A reasonable workflow is to install Ollama first, then filter available models by three criteria that actually determine whether a model is usable for your task: open weights, sufficient context window, and reliable tool-calling support. That last one eliminates more candidates than people expect.
Size the Model to the Hardware
The most common beginner mistake is downloading the biggest model that fits, then concluding local AI is unusably slow.
- Phones: stay at 3B parameters or below.
- Laptops: 7B to 14B is the practical sweet spot.
- Starting point: a 7B GGUF quantised model — DeepSeek-R1 7B Q4 or Llama-3.1 8B Q4 — balances accuracy, speed and memory on most laptops.
- Coding work: DeepSeek Coder V2 is the specialist choice.
- Edge and low-power devices: Gemma 2 at 9B, or smaller quantised variants.
Performance on Apple silicon is now genuinely comfortable. A quantised GGUF export has been reported at roughly 35 to 50 tokens per second on an M4 Pro Mac, with under a second to first token — fast enough that local inference stops feeling like a compromise for interactive work.
On Windows machines, the practical baseline for certified AI PC features is an NPU rated at 40 or more TOPS. At the high end, NVIDIA's DGX Spark offers 128GB unified memory and one petaflop of FP4 compute; it launched at $3,999 and was raised to $4,699 in early 2026 amid memory supply constraints.
The Hybrid Pattern
Pure local inference has a ceiling. Frontier reasoning, long-horizon planning and broad web research still run better in the cloud. The hybrid pattern accepts this and routes accordingly.
Perplexity shipped a concrete implementation on 1 September 2026. Its Hybrid Compute for Mac lets a single agent split work between frontier cloud models and smaller open-weight models running on Apple silicon, handing off the confidential portion of a task mid-stream without restarting or losing context. Architecturally it behaves like a dispatcher: web research, planning and heavy reasoning stay in the cloud, while anything touching private files or on-device actions is delegated to a local subagent.
The privacy mechanism is the interesting part. An on-device PII classifier reads each task before it leaves the machine, substituting names, addresses and account numbers with stand-ins and restoring them when the response returns. Perplexity has open-sourced the classifier; its 0.6B routing model, PII-Tracer, reportedly lifts long-context sensitive-data recall from 0.687 to 0.965 on conversations exceeding 10,000 characters. On-device steps consume no cloud credits.
Requirements are specific: Apple silicon, macOS 15 or later, 24GB of unified memory minimum and 32GB for best results, with local models downloaded in one click and no manual runtime setup. The feature is available to opted-in enterprise, Pro and Max subscribers.
Why It Matters
The routing decision is now a design choice you own. For most of the past three years, using capable AI meant accepting that your prompts, and often your documents, left your machine. Hybrid architectures make the boundary explicit and configurable — you decide what crosses it.
That has three concrete consequences. Compliance conversations get easier when you can demonstrate that regulated data never left the device. Costs drop, because routine steps that once burned API credits now run locally for free. And offline reliability improves for the subset of work the local model can handle alone.
The honest caveat: hybrid routing adds a failure mode. When a dispatcher decides what is sensitive, a classification miss sends data you meant to keep local into the cloud. Open-sourcing the classifier, as Perplexity has done, is the right response — it makes the boundary auditable rather than asking users to trust a black box.
A Sensible Setup for Most People
Install Ollama. Pull a 7B or 8B quantised model. Point your existing tools at localhost and use it for drafting, summarising, code completion and anything touching private files. Keep a cloud API key for the harder reasoning tasks where the quality gap is still real, and make the escalation deliberate rather than automatic.
That configuration costs nothing beyond hardware you already own, keeps the majority of your data on your machine, and preserves access to frontier capability when a task genuinely needs it. The forecast that local-first will win the interactive and privacy-bound layers of the agent loop first looks increasingly like a description of the present rather than a prediction.
