A company cannot send certain data to an external service. It has reports, case files or internal documents that it wants to classify, summarise or turn into structured information. The same question comes up more and more often: can we install an AI model on our own computer and do the work there?
The honest answer is less comfortable. For a bounded task —classifying documents, extracting fields or preparing a first summary— it may be yes. For installing a “private ChatGPT” that can handle almost anything with the same quality and speed on an ordinary laptop, it is usually no.
Between those extremes lies a territory that gets wider every month. Running a model locally does not guarantee that it is good enough, that it responds at a reasonable speed or that the system complies with regulation. But laboratory hardware is no longer required to obtain something useful either.
That territory is moving quickly. Models that would recently have seemed enormous now fit, once quantised, on a consumer GPU. Others, with between four and twelve billion parameters, can perform specific document tasks on a laptop that have genuine business value.
Two machines announced by Apple in August make the breadth of that term quite clear. The M6 Mac mini offers up to 32 GB of unified memory; the M5 Ultra Mac Studio, up to 512 GB and 1.2 TB/s of memory bandwidth. Both are presented as machines for on-device AI, but they do not address the same class of problem. One can work with small and medium models. The other can keep models with hundreds of billions of parameters in memory. The comparison shows why “running AI locally” is still too imprecise a description on which to base an architecture decision.
The question is no longer whether AI can run locally. We now need to ask which task, with what error rate, on what data and at what speed.
What running a model locally actually means
During training, a model learns millions or billions of numerical values called weights. They are the result of its learning: small numerical adjustments that, taken together, determine how it responds to an input. Training a general-purpose model still requires vast infrastructure. Inference is different: it means loading those already learned weights and using them without training the model again.
When we download an open-weight model and open it with tools such as llama.cpp, Ollama or LM Studio, inference can take place entirely on our machine. The text does not have to travel to the model developer’s servers. For a trial, the experience is quite close to installing an application: download a program, choose a model and start a conversation.
But three deployments that are usually bundled together under the word local should be distinguished:
- on-device, when the model runs on one person’s laptop or phone;
- on-premises, when it runs on a server controlled by the organisation;
- in a private cloud, when the infrastructure is remote but isolated and operated under the company’s conditions.
All three can avoid sending data to a public API, but they do not provide the same guarantees. A local application may retain histories, download components, send telemetry or synchronise folders to an external backup. The model can be in the office while the OCR, vector database or observability system remains outside it.
Local inference describes where the model runs, not the entire route travelled by the data.
A model running does not make the process viable
When evaluating a process that uses local inference, three conditions should be separated:
- Do the model and required context fit within the controlled infrastructure?
- Are latency and concurrency acceptable for the process?
- Does the output meet the task’s quality and error threshold?
Meeting the first condition only proves that the model can be loaded. It does not prove that the system will be useful.
The first can be estimated. A model with 27 billion parameters takes around 54 GB if 16 bits —16 binary digits— are reserved for every weight. Quantisation saves space by representing those values with less detail. It resembles rounding millions of numbers: the model becomes smaller in exchange for losing some precision. At 4 bits, the theoretical figure falls to roughly 13.5 GB, although the real file and runtime require more.
Qwen 3.8 27B illustrates the shift. Its original weights take 55.6 GB. A Q4 version —quantised to roughly four bits per weight— used by Simon Willison is around 17 GB and can analyse images, generate code and ask other tools to perform actions. That does not mean it can run in 17 GB of total memory: the application, operating system, intermediate calculations and context memory all need room too.
The context window deserves particular attention. It is the amount of text the model can keep in mind while answering, measured in tokens: the fragments of words into which it divides text. A model may advertise 128,000 or 256,000 tokens and still be unable to use them sensibly on a laptop. As a conversation grows, it must reserve more memory to remember those fragments and takes longer to read the complete input. The maximum context on the specification sheet is not the operational context of our system.
Latency and the ability to serve several requests depend heavily on memory bandwidth. Capacity determines whether the weights fit; bandwidth determines how quickly the processor can read them. It is the difference between the size of a bookshelf and how quickly books can be taken from it. During generation, the machine repeatedly has to move a large portion of the weights. That is why a GPU with fast memory usually responds much better than a CPU, even when both have enough capacity. Unified memory in Apple Silicon and some AMD processors lets CPU and GPU share the same pool; a dedicated NVIDIA GPU will often be faster, but is limited by its own memory, the VRAM.
llama.cpp can also split a model between CPU and GPU. That may be useful in a trial, but reading weights from RAM across the bus reduces speed and may invalidate the result when a process needs interactive responses or several simultaneous requests.
These figures help establish what kind of trial may make sense in each environment:
| Memory available in the environment | Quantised models that commonly fit | What it lets us evaluate |
|---|---|---|
| 16 GB RAM without a dedicated GPU | 2B–8B; 12B with tuning | Classification, simple extraction and tightly bounded flows. Long documents and concurrency expose the limit quickly. |
| 32 GB RAM or unified memory | 8B–14B with room; 20B–27B with compromises | Document analysis, RAG —retrieving relevant passages from internal documentation before answering— and internal assistants on which to begin a serious evaluation. |
| GPU with 12–16 GB VRAM | 8B–14B entirely on GPU; some MoEs around 20B | Interactive latency with moderate context. A 27B commonly needs more quantisation or RAM offload. |
| GPU with 24 GB VRAM | 20B–32B at 4 bits | More complex instructions, tools and code, while measuring error cost and concurrency. |
| 64–128 GB unified memory or several GPUs | 70B and some larger MoEs with aggressive quantisation | More open cases and difficult exceptions; the quality gain must justify cost and latency. |
| 192–512 GB unified memory or a multi-GPU server | Models above 100B and large quantised MoEs | The technical limit of a workstation, not the starting point for an enterprise architecture. |
Compact machines with 128 GB of unified memory, such as NVIDIA DGX Spark or systems built around the Ryzen AI Max+ 395, can load much larger models. NVIDIA advertises up to 200B parameters and AMD has demonstrated 128B models. These are demonstrations of memory capacity, not promises of an interactive experience: loading a 100B model and wanting to use it all day remain different problems.
Apple’s new range also helps separate capacity from speed. The M6 Mac mini offers up to 32 GB and 170 GB/s; the M5 Pro in the same form factor reaches 64 GB and 307 GB/s. The M5 Ultra Mac Studio goes up to 512 GB and 1.2 TB/s. A newer chip does not remove the physical limit: if the weights do not fit, they do not fit. Apple announced these machines on 25 August 2026 and will begin shipping them on 22 September, so the M5 Ultra figures are still manufacturer specifications rather than independent user experience.
The same capability requires less infrastructure over time
Parameter count stopped being a sufficient measure of intelligence a long time ago. The quality and composition of the data, architecture, post-training, reinforcement learning and the ability to spend additional compute reasoning about a difficult request all matter.
The model generation matters too. Stanford’s AI Index offers a revealing comparison: in 2022, the smallest model scoring above 60% on MMLU had 540B parameters. In 2024, Phi-3 Mini crossed the same threshold with 3.8B, a 142-fold reduction. MMLU does not represent the real work of a company, but the movement of the boundary is meaningful.
An Epoch AI analysis estimated that algorithmic improvements roughly halved every eight months the compute required to reach a given level of performance in pretrained language models. The uncertainty interval is wide, and much of total progress still comes from training with more data and compute. Even so, it helps explain why comparing an 8B model today with an 8B model from two years ago tells us very little.
Efficiency comes through several routes:
- better data and post-training extract more value from each parameter;
- quantisation reduces memory and bandwidth, with a quality loss that can be small or significant depending on the model and task;
- mixture of experts architectures divide a model into specialised groups and consult only some of them for each token, although all must remain loaded;
- speculative decoding lets a cheap mechanism propose several tokens for the main model to verify;
- the programs that run models make better use of GPUs, unified memory and NPUs, chips dedicated to accelerating AI operations;
- specialised models give up being good at everything to solve a specific function with fewer resources.
MoE models create another common misunderstanding. gpt-oss-20b has 21B total parameters but uses about 3.6B to generate each token. It works with one part of the model at a time, but must keep the whole model in memory: its quantised weight file still takes 12.8 GB, and OpenAI puts the minimum system memory at around 16 GB. Google explains the same point for Gemma 4 26B A4B: it uses 4B per token, but all 26B must remain loaded so the system can quickly choose which groups take part.
Which tasks are worth testing
Benchmarks, standardised tests used to compare models, help eliminate options and detect advances. They are less useful for telling us whether a system will extract twenty fields from our reports correctly, preserve the same output format over 10,000 documents or abstain when information is missing.
Real uses reveal an uneven boundary:
| Approximate model class | Reasonable tasks for a trial | Risks to measure |
|---|---|---|
| 2B–4B | Classification, rewriting, simple extraction, autocomplete, local commands and highly repeatable tasks | Long instructions, ambiguity, multi-step reasoning and fidelity over long documents |
| 8B–14B | Summaries, structured extraction, drafts, RAG, questions over documentation, analysis scripts and basic vision | Long tool workflows, rare exceptions and decisions requiring reliable expert knowledge |
| 20B–32B | Capable general assistant, code over bounded repositories, vision, tools and multi-step analysis | Long-running agents, speed on modest hardware and reliability comparable to the best hosted systems |
| 70B or more | Greater robustness on open and complex tasks | Cost, latency, operations, and the fact that they can still produce convincing errors |
These bands overlap. A 4B model tuned for one classification can beat a general 27B model. A large model with a poor quantisation, the wrong conversation template or too little context can behave worse than a smaller model served correctly.
Current models help anchor those ranges; they do not remove the need for evaluation.
Gemma 4 12B needs, according to Google, about 6.7 GB for Q4 weights before adding context and runtime. It can receive text, images and audio, and Google demonstrates local file analysis through Python scripts. Fitting on a 16 GB laptop and performing well in a demonstration does not prove that it will preserve a schema, use tools or handle exceptions with the reliability a process requires.
gpt-oss-20b fits in about 16 GB, is text-only and is aimed at reasoning, structured outputs and tools. Its MoE architecture makes it relatively light in compute for its total size. It can be attractive for controlled processes, although it still needs an integration that implements the tools; downloading the weights does not add search, code execution or file access on its own.
Qwen 3.8 27B is a useful reference for the 20B–32B range. It is multimodal, carries an Apache 2.0 licence, and a quantisation of around 17 GB fits on a 24 GB GPU or a machine with enough unified memory. In Willison’s tests it could inspect a repository, build and test a utility, locate objects in images and drive a coding agent. He measured between 15 and 30 tokens per second —a measure of how quickly the answer appears, not of its intelligence— on an M5 Max MacBook Pro and a DGX Spark. That demonstrates a combination of capabilities; it does not guarantee the reliability of a particular process.
DeepSeek V4 Flash lets us observe the opposite end: the limit of what a high-end workstation can load. It is an MoE with 284B total parameters, around 13B active per token and an advertised one-million-token context window. It is not “slightly larger” than Qwen 27B: it requires far less compute per token than a dense 284B model, but all its experts still have to remain in memory. An experimental 2.44-bit MLX quantisation occupies 92.8 GB and reached roughly 32–36 tokens per second at short context on a 128 GB M5 Max MacBook Pro. The same test measured a noticeable loss against the hosted model, while quantisations closer to four bits are around 150–170 GB. It is a technically relevant demonstration, but depends on very specific machines, quantisations and runtimes. It does not describe an option available on any computer or a foundation an organisation should adopt by default.
The two examples represent different decisions. Qwen 27B shows what is becoming operational with reasonable resources. DeepSeek V4 Flash shows how far a workstation costing several thousand euros can go. Calling both “local AI” does not make them the same architecture decision.
The variation between real tests reinforces the same point. Some users complete workflows involving dozens of tool calls on an RTX 3090; others find significant errors while reviewing code from the same model. That is not noise a benchmark will remove. It is the effect of changing the task, prompt, quantisation, runtime and tools.
The only test that ultimately matters to a business is its own suite of real cases, including rare cases and expensive failures.
An example: analysing medical reports without taking them outside the organisation
Imagine the goal is not diagnosis, but converting heterogeneous reports into reviewable fields: date, speciality, medication, tests performed and references to medical history.
A prototype might involve dragging a PDF into a local application and asking for a summary. An operational system needs to separate responsibilities. First, OCR turns pages that contain only an image into text; the model then returns the fields as JSON, a structured format that the rest of the software can read and validate:
document
↓
local OCR and text extraction
↓
local model → JSON + passages supporting each field
↓
deterministic validations
↓
human review of errors and uncertain cases
↓
organisation's system of recordThe system should allow the model to return null when the report contains no value and require it to cite the source passage. Dates, codes and formats are then validated with conventional rules that always produce the same result from the same data. Uncertain cases are routed to review using external checks, not merely the confidence with which the model writes its answer. A representative sample makes it possible to measure accuracy and, above all, false negatives: information that was present in the report but the system failed to detect.
In this kind of process, an 8B or 12B model may be sufficient if the schema is stable and the documents predictable. Moving to 27B may improve complex instructions, difficult documents and exceptions, but it also reduces speed and concurrency. Sizing the infrastructure before defining the acceptable error reverses the right order: first define the threshold and test the smallest model that meets it.
The example also shows that privacy does not depend on the LLM alone. The OCR, temporary storage, logs, interface, backups and destination of the result all need to be checked.
Running locally reduces one exposure; it grants no legal exemption
If data does not leave controlled infrastructure, one important risk disappears or shrinks: handing information to a third party. The French CNIL recommends considering secure, specialised local solutions when personal data or sensitive documents are entered. It does not say that installing a model solves compliance.
Processing personal data is still processing personal data when it happens without an Internet connection. The organisation still needs a defined purpose, a legal basis, minimisation, retention periods, access control, security and ways to uphold people’s rights. Health data also falls under the special-category regime in Article 9 GDPR. Depending on scale, risk and use, a data protection impact assessment may be required.
Three legal layers must also be separated:
- The model licence. Downloadable weights do not mean an absence of conditions. Qwen 3.8 and Gemma 4 are released under Apache 2.0; gpt-oss combines Apache 2.0 with its usage policy. Other models have community licences, redistribution limits or prohibited-use policies. A quantisation downloaded from a third party must retain and document the appropriate licence.
- Protection of the input data. GDPR applies to the purpose and processing, not to the country where the GPU was made. Histories, embeddings —numerical representations of documents used to search them— and instruction logs can also contain personal data.
- Regulation of the system and its use. The AI Act classifies systems by intended purpose. Summarising documents to assist a person is not the same as recommending treatment or becoming medical software. A system that is a medical device, a safety component or participates in certain decisions may fall under high-risk obligations and sector-specific regulation.
The AI Act exemption for some free and open-source software is not a blanket permission either. It does not cover prohibited, high-risk or transparency-regulated systems in the same way.
For a medical, employment, financial or legal case, the legal review needs to cover the complete system and its purpose. The question is not only “what licence does Qwen have?”, but “what decision does our process make, with which data, and who is accountable for it?”.
Downloaded weights are not a production system
The ease of opening a chat can hide the work that follows.
A business may need authentication, per-user permissions, encryption, network isolation, secret management, queues, usage limits, auditable traces, model updates, regression tests and a rollback procedure. If several people share a workstation, concurrency must also be sized: a machine that responds well to one conversation can degrade when it receives ten.
Open-weight models also tend to arrive with less product around them. A service such as ChatGPT is not just a weight file. It includes search, tools, filters, an execution environment, context management, infrastructure and updates. Locally, somebody must build or choose those pieces and maintain them.
That control is itself part of the value. A team can pin a version, prevent network egress, observe every call and remove capabilities it does not need. But the cost moves from a price per token to hardware, electricity, integration, security and operations.
When it fits the architecture
Local inference usually makes sense when several conditions align:
- the data is sensitive or there are strict residency and control requirements;
- the task is well defined and can be evaluated with real examples;
- a local model reaches the required quality, even if it is not the best available model;
- volume is stable and high enough to use the hardware;
- offline work, lower network latency or pinning a version has value;
- the organisation can operate the system and own its updates.
It makes less sense when use is sporadic, maximum capability is always required, current knowledge is essential or no team exists to maintain the infrastructure. A properly contracted and configured enterprise API can be safer than an improvised local server. Physical location is not a replacement for sound architecture.
Many cases will use a hybrid design. A small model can classify, draft or remove identifying information locally. Difficult cases can move to a larger model in a private cloud or, where legally possible, an external service. The system keeps document retrieval local and escalates only what is necessary.
The decision should not be framed as an ideological choice between cloud and sovereignty. It is an architecture decision based on risk, quality, latency and total cost.
What is actually changing
The most important trend is not that a laptop will run the best model in the world. Frontier models will also continue to grow, use more compute during a response and integrate with systems that no laptop can reproduce.
What matters is that the capability level a company considers sufficient becomes smaller every year.
We will see more models designed for devices from the start: architectures with few active parameters, quantisation included in training, smaller context memories and prediction of several tokens at once. Manufacturers will keep increasing shared memory and bandwidth. Inference programs will become better at using GPUs and NPUs without every project having to develop hardware-specific optimisations.
We will also see fewer generic local chats and more invisible functions inside products: classifying an incident, extracting invoice data, searching documentation, preparing a reply or transcribing and organising a meeting. A small model that solves one task predictably can be more valuable than a general assistant that is impressive in a demonstration.
For businesses, the most interesting effect will be architectural. Some inference will move closer to the data and become one more application layer. Larger models will remain an escalation route for exceptions, just as today we do not send every workload to the same database or service.
Local inference does not remove the hard problems of AI. It changes which one appears first. We no longer need to start by asking whether a machine exists that can run the model. We can start with the question that should always have come first:
which specific part of the process should we entrust to this model, and how will we know when it is wrong?
Sources
- Stanford HAI: Technical Performance, AI Index Report 2025
- Epoch AI: Algorithmic progress in language models
- Qwen: Qwen 3.8 27B official model card and weights
- Simon Willison: practical tests with Qwen 3.8 27B
- LocalLLaMA: Qwen 3.8 27B experiences in real programming work
- LocalLLM: models users keep as daily assistants
- Google: Gemma 4 memory requirements
- Google Developers: Gemma 4 12B on laptops
- OpenAI: introducing gpt-oss
- OpenAI: gpt-oss model card
- llama.cpp: inference and quantisation on consumer hardware
- NVIDIA: DGX Spark specifications
- AMD: running large models locally on Ryzen AI Max+ 395
- Apple: Mac mini with M6 and M5 Pro
- Apple: Mac Studio with M5 Max and M5 Ultra
- DeepSeek: official DeepSeek V4 Flash model card and weights
- MLX Community: 2.44-bit quantisation and M5 Max tests
- CNIL: questions and answers on deploying generative AI
- European Data Protection Board: Opinion 28/2024 on AI models and personal data
- European Commission: navigating the AI Act
- EUR-Lex: General Data Protection Regulation
