Written by Jakub Rusinowski · Last updated September 6, 2026
These are the strongest local models that fit entirely in 4 GB of VRAM, ranked by capability, with the quantization level and estimated tokens/sec needed to fit.
| Gemma 3 — Gemma 3 4B Instruct | Q4_K_M · 2.415 GB · ~99 tok/s on NVIDIA GeForce RTX 5060 |
| Phi-4 Mini — Phi-4 Mini (3.8B) | Q4_K_M · 2.29425 GB · ~103 tok/s on NVIDIA GeForce RTX 5060 |
| Qwen 3.5 — Qwen 3.5 4B | Q4_K_M · 2.415 GB · ~102 tok/s on NVIDIA GeForce RTX 5060 |
| VibeThinker — VibeThinker 3B | Q4_K_M · 1.8655875 GB · ~121 tok/s on NVIDIA GeForce RTX 5060 |
| Phi 3.5 Family — Phi 3.5 Mini | Q4_K_M · 2.29425 GB · ~107 tok/s on NVIDIA GeForce RTX 5060 |
| Gemma 4 (Legacy Listing — Unverified) — Gemma 4 4B | Q4_K_M · 2.415 GB · ~102 tok/s on NVIDIA GeForce RTX 5060 |
| BitNet b1.58 — BitNet b1.58 3B | 1.58-bit · 0.6557 GB · ~196 tok/s on NVIDIA GeForce RTX 5060 |
| StarCoder 2 — StarCoder 2 3B | Q4_K_M · 1.81125 GB · ~136 tok/s on NVIDIA GeForce RTX 5060 |
| EXAONE 3.5 — EXAONE 3.5 2.4B | Q4_K_M · 1.449 GB · ~145 tok/s on NVIDIA GeForce RTX 5060 |
| Aya 3B (Tiny Aya) — Aya 3B | Q4_K_M · 1.81125 GB · ~123 tok/s on NVIDIA GeForce RTX 5060 |
| Qwen 3.5 — Qwen 3.5 2B | Q4_K_M · 1.2075 GB · ~157 tok/s on NVIDIA GeForce RTX 5060 |
| IBM Granite 4.1 — Granite 4.1 3B | Q4_K_M · 1.81125 GB · ~123 tok/s on NVIDIA GeForce RTX 5060 |
| Ministral — Ministral 3B | Q4_K_M · 2.1735 GB · ~110 tok/s on NVIDIA GeForce RTX 5060 |
| Falcon 3 — Falcon 3 3B Instruct | Q4_K_M · 1.9501125 GB · ~120 tok/s on NVIDIA GeForce RTX 5060 |
| SmolLM3 — SmolLM3 3B | Q4_K_M · 1.85955 GB · ~121 tok/s on NVIDIA GeForce RTX 5060 |
or compare on Vast.ai from $0.35/hr (typical low · varies)
As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.
Gemma 3, Phi-4 Mini, Qwen 3.5, VibeThinker, Phi 3.5 Family all fit in 4 GB VRAM.
NVIDIA GeForce RTX 5060, NVIDIA GeForce RTX 4060, AMD Radeon RX 9060 XT 8GB, NVIDIA GeForce RTX 5060 Ti 8GB.