AI & automation · Guide
When should you run an LLM locally instead of using an API?
A local LLM is an open-weight language model that runs on servers the company controls, in its own data center or a private cloud, instead of being called through a provider's API. It makes sense when data cannot leave your environment, when the system must work offline or when volume is high and steady.
Master TimmUpdated on 7 min read
What a local LLM is, in practice
Running a model locally means downloading the weights of an open model and serving it on your own infrastructure. Families such as Llama, Qwen, Mistral, Gemma and gpt-oss publish their weights, each under its own license, and tools such as Ollama, llama.cpp and vLLM turn those weights into a service your application calls.
- In your data center (on-premises): Full control of hardware and network, including with no internet access.
- In your cloud account: GPU servers inside your private network, with no data passing through a model provider.
- On the user's computer: Small models for tests, prototypes and individual use.
The opposite is a model behind an API: Claude, GPT and Gemini run on the provider's or a partner cloud's infrastructure, and the application sends the data on every call. In between, some clouds offer providers' models in a region you choose, which covers part of the data residency requirements.
When a local LLM makes sense
The decision usually starts from a concrete constraint, not a technical preference. The most common signs:
- Data that cannot leave: A contract, professional secrecy or internal policy forbids sending the content to third parties, even with privacy clauses.
- Data residency: Sending personal data to a provider abroad is an international transfer, which Brazil's LGPD allows only under its article 33 mechanisms, such as the ANPD's standard contractual clauses; other laws and contracts set similar limits.
- Offline operation: Plants, ships, remote sites or environments isolated for security (air-gapped).
- Predictable latency: On the same network, the model answers without depending on the internet or the provider's queue, as long as the hardware can handle the load.
- High, steady volume: With heavy use all day, the hardware stays busy, and the cost per task can drop below the per-token cost of an API.
- Version control: The model changes only when you decide, regardless of the provider's deprecation schedule.
Local LLM or API: what changes on each criterion
Neither option wins on every criterion. The table sums up what changes; how much each row weighs depends on your case.
| Criterion | API model | Local LLM |
|---|---|---|
| Data and privacy | Sent to the provider on every call, under its contract and retention policy | Stays on your network, and access control is yours |
| Quality | Frontier models are available only through an API or partner clouds | Models that fit your hardware tend to be smaller: good at well-defined tasks, and in need of testing on complex ones |
| Cost | Per token, in proportion to usage, with no upfront investment | Hardware, power and operations, which cost the same at low or high usage |
| Latency | Depends on the internet and on the provider's load | Depends on your hardware and the number of concurrent users |
| Operations | The provider runs the infrastructure and the updates | Your team runs servers, updates, security and monitoring |
| Scale | Absorbs peaks with no capacity planning | Limited to the GPUs you have; peaks need reserved capacity |
What drives the cost of each option
The two options have different kinds of cost, and the comparison only holds at the same workload and the same quality level.
- API: Input and output tokens multiplied by the calls; nothing costs money while idle, and the bill grows with usage.
- Hardware: Buying or renting GPU servers, sized for peak usage rather than the average.
- Operations: Power and cooling, or the cloud rental, plus the team's time to update, secure and monitor the service.
- Utilization: An idle GPU costs the same as a busy one; a local model pays off when the load is high for much of the day.
- Quality: If the local model fails more often, human review and rework go on the bill.
That is why there is no ready answer to which option is cheaper. The answer comes from measuring: tokens per task and volume on one side; hardware, utilization and operations on the other.
Hardware: what a local model needs
The resource that limits a local model the most is GPU memory. It holds the model's weights and, while in use, the attention cache (KV cache) of every conversation in progress.
- Model size: The memory for the weights grows with the number of parameters and the precision of each one.
- Quantization: Storing weights in 8 or 4 bits reduces the memory needed, with some quality loss that has to be measured.
- Context and concurrency: Long contexts and many simultaneous users take extra memory in the KV cache.
- Inference server: In production, engines such as vLLM serve many users at once with continuous batching; Ollama and llama.cpp make it easy to run models on a single machine.
Sizing comes from a load test with the model, the context length and the real number of users, not from a generic table.
Hybrid architecture: local and API models in one system
The choice does not have to be all or nothing. A routing layer decides, for each request, which model serves it, and the application does not need to know which one did.
- By sensitivity: Personal or confidential data goes to the local model, and the rest can go to the API.
- By difficulty: Simple tasks, such as classifying and extracting, stay on the local model, and complex reasoning goes to a frontier model.
- By availability: If one side fails or hits its limit, the other takes over as a fallback.
Because vLLM and Ollama expose OpenAI-compatible APIs, switching between local and cloud can be a configuration change, as long as every route goes through the same evals. It is the same design as Mirador, The Hackett Group's generative-AI engine: Claude and OpenAI behind a provider abstraction, with runtime model selection.
Related service
AI & LLM Engineering
AI agents, RAG and MCP in production, not in demos.
Free first conversation · Reply within 1 business day
See also: Software Architecture.
Frequently asked questions
Is a local model as good as ChatGPT or Claude?
It depends on the task. On well-defined tasks, such as classifying, extracting data, summarizing or creating embeddings, open models that fit on a company server usually do well; on long reasoning and complex tasks, frontier models behind an API tend to get more right. The way to know is to run both on the same evaluation dataset.
Can you run an LLM on a laptop?
Yes, with small or quantized models, using tools such as Ollama or llama.cpp. It works for tests, prototypes and individual use, but it does not replace a server when many people use the system at once or when the task needs a larger model.
Do privacy laws require running the model locally?
Usually not. Brazil's LGPD, for example, allows personal data to be processed by providers abroad when the international transfer rests on one of its article 33 mechanisms, such as the standard contractual clauses approved by the ANPD, and the law's other principles are met. A local model is one way to avoid the transfer, not a requirement; contracts and sector rules can go further, and the final call belongs to your legal team.
Can we start with an API and move to a local model later?
Yes, if the system is built from the start with a model abstraction layer and an evaluation dataset. The application calls a single interface, and switching models becomes configuration; prompts may need tuning, and the evals show whether the local model keeps the quality before the switch.
Who updates a local LLM when a new version comes out?
Your team, because nothing changes on its own. That is an advantage, since the model changes only when you decide, and also a cost: every new version goes through the evals, the load test and an update of the inference server, which also gets security fixes.
Sources
Master Timm · Updated on