Most guides to local AI start with a machine that costs more than a used car. This one starts with the laptop I actually own: a 2018 ultrabook with 8 GB of RAM and no discrete GPU.
The short answer is yes, you can run a language model on 8 GB of RAM. The useful answer is that you can run some models, at some speeds, for some tasks — and the difference between “this is amazing” and “this is unusable” is almost entirely about memory headroom.
The verdict
A 1B–4B model at Q4 quantization is genuinely useful on this machine. A 7B model fits, but leaves so little memory that opening a browser ruins it. Anything at 13B or above is a cloud job.
The memory math that decides everything
A model's file size is not its memory use, but it is close enough to plan with. The thing that makes 8 GB viable is quantization: storing the model's weights at 4 bits instead of 16.
| Model size | FP16 (unquantized) | Q4_K_M | Fits in 8 GB? |
|---|---|---|---|
| 1.7B | ~3.4 GB | ~1.1 GB | Comfortably |
| 3B | ~6 GB | ~2.0 GB | Yes, with room |
| 4B | ~8 GB | ~2.5 GB | Yes |
| 7B | ~14 GB | ~4.7 GB | Barely |
| 8B | ~16 GB | ~4.9 GB | No margin |
| 13B | ~26 GB | ~7.9 GB | No |
Now subtract what Windows is already using. On a clean boot, this laptop has about 5.5 GB free. Load a 4.7 GB model and you have roughly 0.8 GB left — enough for the model to run, not enough for anything else.
That is the entire story of local AI on low-end hardware: the model fits, the margin does not.
Three limits, in order of importance
1. Capacity
If the model plus the KV cache (the conversation's memory) exceeds free RAM, Windows swaps to disk. Generation speed collapses from “slow” to “why is the fan screaming.” This is a hard wall, not a gradual slowdown.
2. Memory bandwidth
CPU inference is bandwidth-bound. A 15 W laptop with single-channel DDR4 moves far less data per second than a desktop with dual-channel memory or any GPU. This is why a 3B model generates 6–9 tokens per second while the same model on a modern desktop does 30+.
3. Thermals
A thin laptop cannot sustain full CPU load. After about eight minutes of inference, this chassis throttles and sustained speed drops 10–20%. Short bursts are faster than long sessions. A $20 stand that lifts the rear edge helps more than any software tweak.
What actually runs
Measured on the test bench, browser closed, five runs:
| Model | Quant | Size | Speed | Experience |
|---|---|---|---|---|
| Qwen3 0.6B | Q8_0 | 0.7 GB | 25–35 tok/s | Fast, shallow |
| Qwen3 1.7B | Q4_K_M | 1.1 GB | 12–18 tok/s | The surprise winner |
| Llama 3.2 3B | Q4_K_M | 2.0 GB | 6–9 tok/s | Good balance |
| Gemma 3 4B | Q4_K_M | 2.5 GB | 4–6 tok/s | Better writing, slower |
| Qwen2.5 7B | Q4_K_M | 4.7 GB | 2–4 tok/s | Quality, if you can wait |
| Llama 3.1 8B | Q4_K_M | 4.9 GB | 1.5–3 tok/s | Avoid |
The 1.7B model is the one I keep coming back to. It is fast enough to feel like a tool rather than a batch job, and for rewriting a paragraph, naming a function or explaining an error message, the quality gap to a 7B model is smaller than the speed gap.
Install it in ten minutes
I use Ollama because it handles model downloads, quantization selection and a local API without configuration. Install it, open a terminal, and pull a small model:
ollama pull qwen3:1.7b
ollama run qwen3:1.7b
Ask it something you would normally paste into a chatbot:
Explain what a race condition is in three sentences, then give one Python example.
To see the speed on your own hardware, use verbose mode and look at the eval rate line:
ollama run qwen3:1.7b --verbose "Write exactly 50 words about SSDs."
If your eval rate is below 3 tokens per second, drop to a smaller model before blaming the software. If it is above 10, you have room to experiment.
Set these before you judge performance
- Close the browser. Ten tabs can consume 1.5 GB.
- Set Windows power mode to Best performance and plug in AC power.
- Keep the context window short. A 32k context is not free; the KV cache grows with every token.
- Put the laptop on a hard surface. Bed and cushions are thermal traps.
Making 8 GB feel less like 8 GB
Use one model at a time. Ollama keeps a loaded model in memory for five minutes by default. Set OLLAMA_KEEP_ALIVE=0 if you switch tasks often, or leave it if you are having a conversation.
Short prompts, short answers. Prompt processing is the slowest phase on a CPU. Pasting a 3,000-word document and asking for a summary works, but the first token can take 90 seconds. That is fine for a batch task and terrible for chat.
Pick the smallest model that passes your test. Write down three real tasks — rewrite this email, explain this function, suggest a name — and try 1.7B against 4B on those. Most people cannot justify the extra wait.
Do not run a local model and a video call at once. This is not a multitasking machine when a model is loaded.
Where it breaks
- Long documents. A 20-page PDF plus a question will either exceed the context or take several minutes to process.
- Code generation beyond snippets. The model can explain code and write small functions. It will not scaffold an application reliably.
- Image generation. Stable Diffusion locally is out of reach here; use a cloud free tier.
- Anything with “13B” in the name. It may load, but it will swap.
- Battery use. Inference pegs the CPU. Expect roughly half the usual battery life, and a hot lap.
The honest workflow
The setup that survived daily use is not “local AI replaces the cloud.” It is a split:
- Quick, private, offline tasks → 1.7B or 3B locally.
- Long documents, hard reasoning, code review → a cloud model.
- Images, training, anything GPU-shaped → a free cloud GPU tier or a paid API.
That split keeps the laptop responsive and the electricity bill sane. The value of local inference here is privacy, offline access and zero marginal cost — not raw capability.
What would actually change the outcome
In order of impact on this machine:
- 16 GB of RAM. Turns a 3B ceiling into a comfortable 7B, and removes the paging cliff.
- Dual-channel memory. The 8 GB is single-channel; a second stick improves CPU inference bandwidth.
- A better cooling setup. Recovers the 10–20% lost to throttling.
- A newer CPU. Real, but expensive compared with a used mini PC.
If you are deciding between a new laptop and a $40 RAM stick, read the 8 GB vs 16 GB comparison before spending anything.
FAQ
Can I run a 7B model with the browser open?
Not comfortably. The model leaves under 1 GB free, and a browser will push it into swap. Close the browser, or use a 3B model.
Does a GPU-less laptop support CUDA?
No. CUDA requires an NVIDIA GPU. Integrated graphics can help with some workloads via Vulkan or OpenCL, but llama.cpp's CPU path is the reliable option here.
Is Q4 worse than Q8?
Yes, slightly — quantization loses some quality. But Q8 roughly doubles the file size, which is the difference between fitting and not fitting. At 8 GB, Q4_K_M is the correct trade.
How much RAM do I need for a 13B model?
About 12–14 GB free after the OS, so a 16 GB machine is the realistic minimum. On 8 GB it will swap and become unusable.
Keep reading
- The best small LLMs for 8 GB, tested
- GGUF quantization explained
- Ollama vs llama.cpp on this laptop
- The full benchmark table