Here’s a thing that catches almost everyone the first week they run a model locally. You paste a 600-line file into your shiny new local assistant, ask it to find the bug, and it confidently rewrites a function that isn’t even in the part it read. No error. No warning. It just… silently dropped most of your file on the floor before the model ever saw it.
That’s not the model being dumb. That’s Ollama doing exactly what it was told. By default it gives every model a context window of 2048 tokens and quietly truncates anything past that. It’s one of a handful of small surprises that separate “I installed Ollama” from “I actually understand what’s running on my machine.” Let’s go through the ones that matter: how the thing works under the hood, what hardware you really need, the gotchas, and the honest answer to “should I even bother instead of just calling an API?”
What Ollama actually is
Ollama gets described as “Docker for LLMs,” and that’s a decent first approximation. You pull a model, you run it, there’s a registry. But it hides what’s doing the heavy lifting. Underneath, Ollama is a friendly wrapper around llama.cpp, the C/C++ inference engine that made running these models on consumer hardware practical in the first place. When you type ollama run, you’re really booting a llama.cpp runtime with a sane default config and a tidy HTTP server bolted on.
The models it runs are in a format called GGUF (GPT-Generated Unified Format). A GGUF file isn’t just weights. It’s a self-contained package that bundles the tensors, the tokenizer config, the architecture details, and hyperparameters like the trained context length, all in one file. That’s why ollama pull llama3.1 gives you something that just works: everything the runtime needs to reconstruct the model is in the box.
Ollama itself is young. The project shipped its first release in early July 2023, and it rode the wave of open-weight models (Llama 2 landed that same month) that suddenly made “run a real LLM on your laptop” a thing normal developers could do. Before that, local inference meant compiling things and reading a lot of GitHub issues. Ollama’s whole pitch is removing that friction.
The hardware math nobody explains up front
The number that decides whether a model runs well on your machine isn’t its parameter count. It’s how much memory the weights occupy after quantization. This is the single most important concept for running models locally, so it’s worth slowing down for.
A model’s weights are originally stored in 16-bit floating point. Quantization squeezes them down to a lower precision, commonly 4-bit integers, which shrinks the file and, just as importantly, eases the memory-bandwidth pressure that bottlenecks inference. The format you’ll see by default in Ollama is Q4_K_M, part of llama.cpp’s “K-quant” family. The trade is genuinely good: Q4_K_M cuts memory use by roughly 75% versus the 16-bit original while losing well under 1% of quality on most benchmarks. That’s not a free lunch exactly, but it’s close enough that most people never run anything else.
Here’s the rule of thumb that actually helps you size hardware: budget about 0.6 GB per billion parameters at Q4_K_M, then add headroom for context. So:
Model size
Q4_K_M footprint
Fits comfortably on
7B
~4-6 GB
8 GB GPU, or any M-series Mac
13B
~8-10 GB
12 GB GPU
32B
~22-24 GB
RTX 4090 (24 GB)
70B
~38-48 GB
2x 24 GB GPUs, or a 64 GB Mac
The memory you want this to live in is VRAM, your GPU’s memory, because that’s where inference is fast. If the model doesn’t fit in VRAM, Ollama will happily run it on the CPU using system RAM instead, and it’ll work, just slowly. On Apple Silicon the line blurs in a nice way: unified memory means the GPU and CPU share one pool, so a 64 GB Mac can run models that would need multiple discrete GPUs on a PC.
What does this buy you in speed? Be realistic about it. On CPU-only inference you’re looking at roughly 10-25 tokens per second, usable for short answers, painful for long ones. Put the same model fully on a decent GPU and you jump to 40-80+ tokens/sec; an RTX 4090 can hit 130-160 tokens/sec, which is in the same league as a cloud API. The hardware is the whole game here. A local model on the wrong hardware isn’t a cheaper API, it’s a worse one.
The silent context-window trap
Back to the gotcha from the opener, because it’s the one that wastes the most hours. Ollama defaults num_ctx, the context window, to 2048 tokens for every model, regardless of what that model was actually trained to handle. Llama 3.1 supports 128k tokens of context; out of the box, Ollama gives it 2048.
This default is deliberate, not a bug. It lets Ollama boot any model instantly on any hardware, including an 8 GB laptop, without forcing you to calculate your memory budget first. The problem is what happens when you exceed it: Ollama silently clips the input. No error, no warning. The tokens past your limit simply never reach the model. If you’ve ever fed a local model a big file and watched it “forget” the beginning, this is almost always why.
You fix it in one of two places. For a one-off, pass num_ctx in the request options:
Per-request override
curl http://localhost:11434/api/generate -d ‘{
“model”: “llama3.1”,
“prompt”: “Summarize this file…”,
“options”: { “num_ctx”: 16384 }
}’
Enter fullscreen mode
Exit fullscreen mode
For a permanent per-model default, bake it into a Modelfile and create your own variant:
Modelfile
FROM llama3.1
PARAMETER num_ctx 16384
Enter fullscreen mode
Exit fullscreen mode
Build it once
ollama create llama3.1-16k -f Modelfile
ollama run llama3.1-16k
Enter fullscreen mode
Exit fullscreen mode
But there’s a cost, and it’s not optional: the context window lives in the KV cache, and that grows linearly with num_ctx. Bumping a 7B model to a 32k window can add around 6 GB of VRAM on top of the weights. So context length isn’t a free dial you crank to maximum. It competes directly with the model for the same memory. Pick the smallest window that fits your actual workload.
WarningThe 2048 default plus silent truncation is the single most common reason people conclude “local models are dumb.” They’re usually not. They’re just being shown a fraction of the input. Check your num_ctx before you blame the model.
Wiring it into your editor
The reason most developers reach for this in the first place is a private coding assistant: autocomplete and chat that never sends a line of your code anywhere. Ollama exposes a local HTTP API on port 11434, and editor extensions like Continue talk to it directly. Your code goes from your editor, to a process on your own machine, and back. Nothing crosses the network.
The wiring is small. Point your Continue config at the local model:
Continue config (shape may vary by version)
{
“models”: (
{
“title”: “Llama 3.1 8B (local)”,
“provider”: “ollama”,
“model”: “llama3.1:8b”
}
)
}
Enter fullscreen mode
Exit fullscreen mode
That’s the whole privacy story, and it’s a real one: with the model pulled, you can pull the ethernet cable out and it keeps working. Ollama doesn’t phone home during normal inference: no telemetry upload, no cloud sync, no prompts shipped to a third party. The model files sit on your disk until you delete them, and only the initial ollama pull needs the internet. For anyone working under HIPAA, PCI-DSS, or GDPR data-residency rules, that’s not a nice-to-have. It’s frequently the only arrangement that’s even allowed, because no amount of vendor paperwork beats the data physically never leaving your machine.
The memory-management gotcha
One more behavior worth knowing before it confuses you. After you finish a request, Ollama keeps the model loaded in VRAM for 5 minutes by default, so your next prompt answers instantly instead of paying the load cost again. Handy, until you’re trying to run a second large model and discover the first one is still squatting on your GPU memory.
You control this with keep_alive. Set it to 0 to unload the moment a response finishes, or to something like “24h” to pin a model in memory all day:
Unload immediately after responding
curl http://localhost:11434/api/generate -d ‘{
“model”: “llama3.1”,
“prompt”: “quick question”,
“keep_alive”: 0
}’
Enter fullscreen mode
Exit fullscreen mode
You can check what’s currently resident with ollama ps and evict a model by hand with ollama stop. If you’re juggling several models on a memory-tight machine, managing keep_alive is the difference between smooth switching and constant out-of-memory errors.
When local actually beats an API
Now the honest part, because the answer isn’t “always.” Running locally is a real engineering trade, and plenty of the time the cloud is just the better call.
Cost is the trap people get wrong in both directions. The rough crossover: under about 1M tokens a day, a cloud API is usually cheaper once you account for the hardware you’d have to buy and run. Past roughly 5M tokens a day, owning the hardware starts paying for itself. Below that line, a $1,600 GPU sitting mostly idle is a worse deal than per-token pricing. Buying a 4090 to occasionally autocomplete is a hobby, not a saving.
Latency can favor local, especially for short, frequent calls where the network round-trip dominates. But only if your hardware keeps up. Remember the numbers: a top GPU matches cloud throughput, CPU-only inference is 4-10x slower. Local isn’t automatically faster. It’s faster when the GPU is there.
Capability still favors the cloud at the top end. The biggest frontier models you reach through an API are stronger than anything you’ll fit on a single machine. For routine work (autocomplete, summarizing, boilerplate, straightforward refactors) a good local 8B or 32B model is more than enough. For genuinely hard reasoning, the gap is still real.
Privacy and compliance is where local stops being a preference and becomes a requirement. If your data legally can’t leave a boundary (patient records, payment data, regulated EU data) then keeping inference on hardware you control isn’t a tradeoff, it’s the entire point. No enterprise agreement substitutes for the data simply never being transmitted.
The pattern a lot of teams land on isn’t all-or-nothing. It’s a blend: local models for the private, high-volume, latency-sensitive, offline work, and a cloud API for the occasional heavy request that needs the strongest model available. You don’t have to pick a side. You have to know which job each tool is actually good at.
So start small. Pull an 8B model, point your editor at it, write some real code through it for a week, and watch your token meter not move. Then decide what’s worth keeping local, now that you know what’s actually running on your machine, and why.
Originally published at nazarboyko.com.
Source link
