Local language models are a practical option when you need more control over data flows, operation and model choice. Whether they fit your business depends on your tasks, available hardware and the work required for maintenance and quality checks.
This article reviews the model families named in the original May 2026 overview. Corrections were checked on 30 September 2026. It is not a complete ranking of every model released since then. The original speed figures lacked a documented test configuration and should not guide a hardware purchase.
The Hardware Reality#
Three routes are useful: CPU with sufficient RAM, NVIDIA GPUs with dedicated VRAM, and Apple Silicon with unified memory. CPU core count alone does not determine speed. Memory bandwidth, model, quantization, context length, backend and concurrent requests also matter.
CPU only: A smaller quantized model may be enough for drafts, summaries and background jobs. Measure response times on your own tasks. A particular Intel i9 or Ryzen 9 does not automatically have 32 CPU cores. A large model may fit in RAM while remaining too slow for interactive work.
Consumer GPU: An RTX 4090 has 24 GB VRAM, an RTX 4080 16 GB. A 32B model in a common four-bit format may fit on a 4090 depending on the file and context budget. That is not a promise for the 4080. Weights, KV cache and runtime buffers must fit together.
At a theoretical four bits, 70B weights alone require about 35 GB; at three bits, about 26.25 GB, before overhead. Such a model cannot fit entirely in a 24 GB card's VRAM. RAM offloading or multiple GPUs may make it runnable but substantially change speed and operation.
Apple Silicon: CPU and GPU share memory. This can support models larger than a GPU with limited VRAM. The operating system and other applications need memory too. A 64 GB Mac does not establish a particular token rate or a universal advantage over NVIDIA.
Before buying hardware, test quality, latency, long context and concurrent users on a suitable system. Include electricity, setup and maintenance in costs.
The Models That Matter#
Qwen3: The original release includes dense 0.6B, 1.7B, 4B, 8B, 14B and 32B models, plus 30B-A3B and 235B-A22B MoE models. An original Qwen3 7B or 72B is not part of that family. Smaller variants and 14B are possible first test candidates, not universal winners.
Llama 3.3: The official model meant here is Llama 3.3 70B Instruct. The earlier Llama 3.3 8B recommendation was incorrect. A smaller Llama from another generation must be evaluated under its actual name.
Gemma 3: The original family includes 1B, 4B, 12B and 27B. The previously listed original 8B variant was incorrect. Check the concrete model card, licence, language quality and image support where relevant.
Mistral, Phi-4 and distilled DeepSeek-R1 variants are further candidates. Reasoning models may generate longer answers and take more time. Results on someone else's benchmark do not replace testing your texts, documents or tool calls. Open weights also come with licence terms.
For MoE, active parameters describe computation per token, not total memory requirements. A 235B-A22B model still needs its expert weights available; it is not a 22B model for hardware planning.
The Stack#
Ollama offers an accessible entry point with a local API. llama.cpp gives more direct control over GGUF files and runtime options. vLLM is interesting for serving concurrent requests, subject to model and hardware support. LocalAI combines multiple backends. MLX-LM is an option for Apple Silicon.
Choose based on required interfaces, supported models and operational needs. An OpenAI-compatible API makes integration easier but does not guarantee identical behaviour for every tool.
Quantization and Memory#
Quantization reduces weight storage. It does not make a large model arbitrarily small without quality loss. Q4_K_M can be a reasonable starting point; Q5, Q6 and Q8 require more memory. Compare IQ formats and very low bit widths on your tasks.
Downloaded file size is only part of the calculation. Long context and concurrent requests increase requirements. Successfully loading a model does not mean it will run reliably at your intended context length.
Picking Your Setup#
Start with a manageable model and a documented configuration. Compare candidates on the same tasks: a summary, a draft, a document question and a tool call if needed. Record errors, latency and memory use.
For a local Ollama test, you can try ollama run qwen3:14b. Check the variant and quantization actually loaded. Qwen3 8B is a smaller candidate for a less powerful machine. A team server additionally needs a load test and access controls.
Local Means Checking Data Flows#
Local inference can keep prompts from a model provider, but only for the genuinely local path. Cloud models, web search, embedding APIs, remote administration, logs and backups may still transfer data. Ollama documents OLLAMA_NO_CLOUD=1 for disabling its cloud features. Check the surrounding application and integrations too.
Future releases may improve quality or change hardware requirements. Purchase decisions should follow reproducible tests of your tasks rather than a forecast for the next quarter.
Sources and review date#
Reviewed on 30 September 2026.
