Running Local LLMs on Unraid
The first local model I ran took about 45 seconds to respond to a simple question. I almost gave up right there. The model was too big for my VRAM, the quantization was wrong, and I hadn’t understood yet that “running a local LLM” and “running a usable local LLM” are meaningfully different things.
I run Ollama inside a Docker container on Unraid. The container gets passed through to an NVIDIA GPU, which handles all the inference. The host is a Ryzen 9 5900X on an ASUS TUF Gaming X570-Plus (Wi-Fi) board. Getting GPU passthrough working was the first real obstacle. Unraid handles it through its Docker GPU assignment settings, but you have to make sure the NVIDIA container toolkit is installed and the container is configured to use it. There are Unraid forum threads on this; I won’t reproduce the whole process here, but it took me longer than I expected because the error messages weren’t obvious about what was actually wrong.
The hardware question matters more than the model question, at first. Most consumer GPUs in the 8-12GB VRAM range can run 7B or 8B parameter models fully in memory and respond in a few seconds. That’s where I’d start. The models in that range, Llama 3, Mistral, Gemma, Qwen, have gotten good enough over the past year that the quality is genuinely useful for most everyday tasks. Quantized versions, typically Q4 or Q5, cut memory requirements significantly with a modest quality tradeoff. For anything that’s not critical reasoning, the tradeoff is worth it.
What I actually run daily is a quantized Llama 3 8B for quick, low-stakes queries and a mid-size model in the 13-20B range for tasks where quality matters more than speed. The 8B model responds in 2-4 seconds on my hardware, which feels fast enough to be comfortable. The larger model takes 15-25 seconds per response, which I find acceptable for deliberate tasks but too slow for conversational back-and-forth.
The models I don’t run locally are the ones where I genuinely need top-tier reasoning or writing quality. For those I send API calls to Anthropic. The local models are good for information retrieval, summarization, drafting, code explanation, and routine agent tasks. They’re not good enough to replace a frontier model for complex multi-step reasoning or polished writing that goes on the internet. Knowing that line has saved me a lot of frustration.
Unraid’s container management makes it fairly easy to keep Ollama updated and to pull new models. The Ollama API makes it straightforward to switch which model a request goes to. The practical challenge is storage: model weights pile up fast. A handful of models can eat 50-100GB without trying hard. I keep the ones I actually use and delete the rest. The Unraid array handles the storage; I’ve dedicated a Samsung 860 EVO 1TB as a cache SSD specifically for models so loading times stay fast.
One thing I wish I’d known earlier: the model name matters, but the quantization label matters almost as much. “Llama 3 8B” can mean very different performance depending on whether it’s Q2, Q4, Q5, or Q8. I run Q4 as a default, Q5 when I want a bit more quality on a task that warrants it. Q8 is overkill for most things and Q2 degrades quality more than I find acceptable.
If you’re curious about specific model recommendations for your VRAM budget, feel free to drop the specs in a comment and I’ll give you my current take.
Hardware linked in this post:
Affiliate disclosure: Some links in this post are Amazon affiliate links. If you buy through them, I get a small commission at no cost to you. It helps keep the lights on here.
