Knowledge Base · Local AI
How much RAM do you need to run a local LLM?
By Tiny · how I pick
As an Amazon Associate I earn from qualifying purchases. The graphics card links below are affiliate links.
A model at the usual 4-bit setting needs about 0.5 to 1 GB of memory per billion parameters, plus 2 to 3 GB of overhead. In practice:
- 7B model: about 6 to 8 GB free
- 13B model: about 10 to 12 GB
- 34B model: about 20 to 24 GB
- 70B model: around 40 to 48 GB
That memory can be regular system RAM or the VRAM on a graphics card. Either one works as long as the model fits. The difference is speed, which I cover further down.
Words you'll see in this guide
- Parameters (7B, 13B, 70B): a rough measure of a model's size. "8B" means 8 billion. Bigger models are usually smarter and need more memory.
- RAM: your computer's main working memory. More RAM lets it run more apps at once. AI models can run from it too, just slower than from VRAM.
- VRAM: the memory on your graphics card. AI models load into it, so more VRAM means you can run bigger models.
- Quantization: shrinking a model by storing its numbers less precisely, so it fits in less memory with a small drop in quality. Think of saving a photo as a smaller JPEG. "4-bit" (Q4) is the common setting.
- Context length: how much text the model can keep in mind at once, measured in tokens. Longer context uses more memory.
- Unified memory: Apple's design where the processor and graphics share one fast pool of memory, so a Mac's RAM works like VRAM.
Memory by model size
| Model size | Memory needed (4-bit) | Runs on |
|---|---|---|
| 3B | 3 to 4 GB | Any laptop with 8 GB of RAM |
| 7B and 8B | 6 to 8 GB | 16 GB of RAM, or an 8 GB graphics card |
| 13B and 14B | 10 to 12 GB | 32 GB of RAM, or a 12 GB card like the RTX 3060 12 GB |
| 34B | 20 to 24 GB | A 24 GB card like the RTX 3090 or RTX 4090, or a 32 GB Mac |
| 70B | 40 to 48 GB | Two 24 GB cards, or a 64 GB Mac |
Two settings that change the number
Quantization. The table assumes 4-bit, which is what almost everyone uses at home. It cuts memory to about a quarter of full precision, and most people can't tell the difference in answers. Run the same model at 8-bit and you need roughly twice the memory. At 16-bit, roughly four times. Unless you have a specific reason, stay on 4-bit.
Context length. The model also needs working memory, called the KV cache, for whatever text it's holding in mind. A short chat adds a couple of GB. A long 32K-token context can add more than 10 GB on top of the model itself. If you plan to paste in long documents, leave extra room.
Does it matter whether it's RAM or VRAM?
For whether the model runs at all, no. It only needs enough of one or the other.
For speed, it matters a lot. A model sitting in a graphics card's VRAM writes text several times faster than the same model running from system RAM on the processor. That's why a 12 GB graphics card beats 32 GB of plain RAM for a 13B model, even though the model technically fits in both.
Macs are the exception. Apple Silicon's unified memory is both fast and large, so a Mac's RAM behaves a lot like VRAM. More on that in Mac vs PC for local AI.
Check your own machine
Put your RAM and your processor or graphics card into the Server Calculator. It tells you which model sizes fit and roughly how fast each one will run.
- Not sure your laptop can do this at all? Start with can my laptop run local AI.
- Shopping for a graphics card? See the best GPU for local AI under $500.
- Thinking about a new build while RAM prices are high? Read is it a good time to build a PC.
Quick answers
How much RAM do you need to run a local LLM?
At the usual 4-bit setting, roughly 0.5 to 1 GB per billion parameters plus 2 to 3 GB extra. So a 7B model needs about 6 to 8 GB free, a 13B model about 10 to 12 GB, a 34B model about 20 to 24 GB, and a 70B model about 40 to 48 GB. That memory can be system RAM or graphics card VRAM, which is much faster.
How much RAM for a 7B model?
About 6 to 8 GB of free memory at 4-bit. A 16 GB machine runs a 7B model comfortably, with room for your system and other apps.
How much VRAM for a 13B model?
About 10 to 12 GB at 4-bit. A 12 GB card like the RTX 3060 12GB fits a 13B model, and a 16 GB card leaves room for longer chats.
Does context length affect how much memory I need?
Yes. A longer context, meaning how much text the model keeps in mind, adds memory on top of the model itself, from a couple of GB for short chats to well over 10 GB at very long contexts.
Free email updates
New models come out every month. Subscribe and I'll tell you which ones fit the memory you already have.
No spam. Unsubscribe anytime. Privacy policy.