DEV Community

Cover image for Running a Local LLM on an Older Computer: A Simple Home Lab Guide
The AI Pal
The AI Pal

Posted on AI-assisted

Running a Local LLM on an Older Computer: A Simple Home Lab Guide

#ai #homelab #beginners #tutorial

Comments highlight llama.cpp and legal text tests

Table of Contents

Why Run an AI Model at Home?

When most people hear "AI language model," they imagine expensive servers and powerful computers.

That is not always necessary.

You can run a small language model on an older laptop, a regular desktop, a mini PC, or a home lab server. It may not be as fast as the largest online AI tools, but it is still useful for learning and experimenting.

In this guide, I will show you how to run a local AI model using Ollama.

What Is a Local LLM?

LLM stands for Large Language Model. It is the technology used by many AI chat tools.

A local LLM runs directly on your computer instead of on a remote server owned by a company.

This has several advantages:

  • Your questions and files can stay on your computer
  • You can use it without paying for every request
  • It can work without an internet connection after the model is downloaded
  • You can experiment freely
  • It is a good way to learn how AI applications work

There are also some limitations. Smaller local models may not be as accurate or detailed as the largest cloud-based models. For a home lab, however, they are often good enough.

What Kind of Computer Do You Need?

You do not need a new gaming computer to get started.

A practical starting system might have:

  • 8 GB of RAM for very small models
  • 16 GB of RAM for a more comfortable experience
  • Four or more CPU cores
  • At least 10 GB of free storage
  • A graphics card is helpful, but not required

If you have an older computer, start with that. You can always move the setup to a more powerful machine later.

The amount of memory is usually more important than having the newest processor. Larger models need more memory, while smaller models can run on ordinary hardware.

Installing Ollama

Ollama is a tool that makes it easier to download and run local AI models.

Download and install it for your operating system. Ollama supports Windows, macOS, and Linux.

After installing it, open PowerShell, Terminal, or Command Prompt and run:

ollama --version
Enter fullscreen mode Exit fullscreen mode

If you see a version number, Ollama is installed correctly.

You can also explore the Ollama GitHub repository if you want to learn more about how it works.

GitHub logo ollama / ollama

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

ollama

Ollama

Start building with open models.

Download

macOS

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

or download manually

Windows

irm https://ollama.com/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

or download manually

Linux

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Get started

ollama

You'll be prompted to run a model or connect Ollama to your existing agents or applications such as Claude Code, OpenClaw, OpenCode , Codex, Copilot, and more.

Coding

To launch a specific integration:

ollama launch claude

Supported integrations include Claude Code, Codex, Copilot CLI, DeepSeek Harness, Droid, and OpenCode.

AI assistant

Use OpenClaw to turn Ollama into a personal AI assistant across WhatsApp, Telegram, Slack, Discord, and more:

ollama launch openclaw

Chat with a model

Run and chat with Gemma 4:

ollama run
…
View on GitHub

Downloading Your First Model

For an older or modest computer, start with a small model:

ollama pull llama3.2:3b
Enter fullscreen mode Exit fullscreen mode

This downloads the model to your computer. The download may take a few minutes, depending on your internet connection. You only need to download it once.

The 3b in the model name means that it has around three billion parameters. You do not need to understand the technical details yet. The important thing to remember is that smaller models are easier for regular computers to run.

Starting a Conversation

After the model has finished downloading, start it with:

ollama run llama3.2:3b
Enter fullscreen mode Exit fullscreen mode

You can now type questions directly into the terminal.

For example:

Explain Docker to me as if I have never used it before.
Enter fullscreen mode Exit fullscreen mode

You can also ask the model to explain error messages, help write small scripts, or summarize text.

When you are finished, press Ctrl + D to exit.

That is all it takes to run your first local AI model.

What If the Model Is Too Slow?

If your computer becomes slow or the model takes a long time to respond, try a smaller model:

ollama pull phi3:mini
ollama run phi3:mini
Enter fullscreen mode Exit fullscreen mode

Smaller models normally use less memory and respond faster. The trade-off is that their answers may be shorter or less detailed.

A simple way to think about it is:

  • Smaller models are faster and easier to run
  • Larger models usually give better answers but need more memory

There is no need to start with the largest model available. A small model is perfectly fine for learning.

Creating a Home Lab Assistant

Ollama lets you create a model with your own instructions.

Create a file named Modelfile and add this:

FROM llama3.2:3b

SYSTEM """
You are my home lab assistant.
Explain things clearly and avoid unnecessary technical language.
When giving commands, explain what each command does.
Warn me before suggesting commands that could delete or change data.
"""
Enter fullscreen mode Exit fullscreen mode

Now create the custom model:

ollama create homelab-assistant -f Modelfile
Enter fullscreen mode Exit fullscreen mode

Start it with:

ollama run homelab-assistant
Enter fullscreen mode Exit fullscreen mode

You now have an assistant designed to help with home lab topics.

It can help explain Linux commands, Docker containers, networking concepts, Python scripts, and server errors. It will not always be correct, but it can be a useful learning companion.

Using Ollama from Another Program

Ollama also provides a local API. This allows your own scripts and applications to communicate with the model.

The API is usually available at:

http://localhost:11434
Enter fullscreen mode Exit fullscreen mode

For example, you can send a request using curl from a Linux or macOS terminal:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Explain what a reverse proxy does",
  "stream": false
}'
Enter fullscreen mode Exit fullscreen mode

This makes it possible to build small projects such as:

  • A private chatbot
  • A log file assistant
  • A documentation helper
  • A command-line question tool
  • A local coding assistant

You do not need to build an application immediately. It is enough to know that Ollama can connect to other programs when you are ready.

Adding a Web Interface

The terminal works well, but a web interface can be more comfortable.

Tools such as Open WebUI, LibreChat, and AnythingLLM can connect to Ollama and provide a browser-based chat interface.

For example, Open WebUI can be started with Docker:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
Enter fullscreen mode Exit fullscreen mode

After it starts, open this address in your browser:

http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

Docker commands can vary slightly between operating systems. Check the Open WebUI documentation if the connection does not work immediately.

Common Problems

The Model Runs Slowly

This is normal when running a model using only the computer's processor.

Try closing other applications and switching to a smaller model. You may also get better performance by adding more RAM or using a supported graphics card.

For simple questions, slower responses are usually acceptable.

Your Computer Runs Out of Memory

This normally means that the model is too large for your system.

Try a smaller model and avoid running many other applications at the same time.

The Answers Are Incorrect

Local models can make mistakes. They may sound confident even when the information is wrong.

Be especially careful with answers about security, medical topics, legal matters, and commands that can delete or change data.


Security note: Use a local LLM as a learning assistant, but verify important information before acting on it. Be especially careful with commands that modify or delete files.

The Model Does Not Follow Instructions

Try giving it more information.

Instead of writing:

Fix this.
Enter fullscreen mode Exit fullscreen mode

Try:

I am new to Linux. Explain why this command failed and show me a safe way to test the fix.
Enter fullscreen mode Exit fullscreen mode

Clear prompts usually produce better answers.

Home Lab Project Ideas

Once everything is working, try building a small project.

Some ideas include:

  • A chatbot that explains server logs
  • A private assistant for your home lab documentation
  • A tool that summarizes text files
  • A script that explains Linux commands
  • A helper for learning Python
  • A question-and-answer tool for your personal notes

These projects do not need to be complicated. The goal is to understand how the different pieces work together.

Keep Your Setup Private

Even though the model runs locally, you should still be careful about how you expose it.

Do not make the Ollama API publicly available unless you understand the security risks. For a home lab, it is usually safest to keep it available only on your computer or local network.

It is also a good idea to:

  • Use a password for web interfaces
  • Keep your operating system updated
  • Keep Docker updated
  • Check unfamiliar commands before running them
  • Back up important configuration files

What Is Coming Next?

In my next post, I will explore how to run larger LLMs on a modest GPU with limited VRAM.

Having a smaller GPU does not necessarily limit you to tiny models. With compressed model formats, the right settings, and a few practical techniques, you can run surprisingly capable open models without investing in expensive hardware.

I will cover:

  • How to choose a model that fits your available VRAM
  • Practical ways to reduce memory usage
  • How to split the workload between GPU VRAM and system RAM
  • The balance between speed, memory usage, and answer quality
  • Some capable open models available for home lab use

The goal will remain the same: to keep the setup affordable, practical, and easy to follow.

If you want to get more from a modest GPU, follow me on DEV so you do not miss the next guide.

Final Thoughts

Running a local LLM is a useful and affordable home lab project.

You can start with an older computer, a small model, and Ollama. The experience may not be exactly the same as using a large online AI service, but that is part of the appeal.

You get to experiment, learn how local AI works, and keep your data close to home.

Start with a small model, try a few simple projects, and upgrade your hardware only when you understand what you actually need.

Top comments (6)

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO •

Really interesting! This is actually exactly what I did a few months ago: I wanted to determine which model performed best depending on the task at hand.

In my case, the task was analysing and summarising decisions from the French Court of Cassation. I ran the same prompt across a range of local models, and the results varied dramatically — from completely unusable to surprisingly excellent.

Interestingly, the best-performing models weren't necessarily the largest ones. It was a great reminder that model size alone isn't a reliable indicator of performance; choosing the right model for the specific task matters a lot.

This is actually one of the building blocks of a legal analytics system I'm working on. For that kind of use case, running the models locally is particularly interesting because it makes it much easier to preserve confidentiality and keep sensitive legal data under control.

Picked as gem
Collapse
 
ai_pal profile image
The AI Pal •

That is a really good real-world example, Pascal. Legal decisions are exactly the kind of material where a model can produce a convincing summary while still missing something important.

Testing the same prompt across several models sounds far more useful than choosing one based only on its size or leaderboard position. Keeping sensitive documents on local hardware is also a strong reason for taking this route.

Which model surprised you the most during your testing, either positively or negatively?

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO • • Edited

Yes — and the results were actually quite interesting.

I tested Microsoft Phi-4, Qwen 3 30B A3B, and Llama 3.3 70B, among others, using exactly the same prompt. The task was to analyse French Court of Cassation decisions: extract the key legal concepts, rank them by relevance, justify each one with an exact quotation from the decision, produce a structured synthesis, identify the legal references, and return everything as strict JSON.

I used analyses produced by Claude Sonnet 4.5 and ChatGPT (the production version available in ChatGPT at the time) as reference points for the evaluation.

The two models that really surprised me were Phi-4 and Qwen 3 30B A3B. Both performed exceptionally well, with Phi-4 slightly ahead — but it was very close.

Llama 3.3 70B, on the other hand, was surprisingly disappointing. It tended to be verbose and quite inconsistent between runs, and it didn't always respect the JSON output requirement. I tested it locally with fairly aggressive quantization, but also tested the unquantized model through DeepInfra. The difference wasn't significant enough to explain the results by quantization alone.

So, in this particular task, the 70B model was clearly not the winner. The smaller models were not only good enough — they were actually more consistent and more useful.

That experiment is one of the reasons I became quite interested in local LLMs for legal analytics: performance really needs to be evaluated against the actual task, rather than inferred from parameter count or general benchmarks. And for legal data, being able to keep the documents and processing local is an additional, rather important, advantage.

Collapse
 
xulingfeng profile image
xulingfeng •

This is an excellent article. My main laptop has an RTX 4070S with 8 GB VRAM and 32 GB RAM. I have tried Ollama, but eventually switched to llama.cpp. It uses fewer resources and gives me much finer‑grained control over parameters, plus it gets updated really often. Ollama felt a little slower in comparison.

I’ve spent a lot of time tinkering on this laptop, with my agent Daoma helping out. Together we have tested almost all the popular small‑size models that fit within 8 GB of VRAM. For my use case, the model that strikes the best balance between speed and quality, and runs the smoothest, is gemma‑4‑e4b‑qat (gemma‑4‑E4B‑it‑qat‑UD‑Q4_K_XL). There will surely be better small models down the line, and I plan to try anything that can fit inside 8 GB VRAM.

Part of my motivation for constantly testing local small models comes from a deep‑seated worry that cloud‑hosted models might suddenly become unavailable. My agent Daoma connects to DeepSeek in the cloud, paired with a local memory system we built together. Compared with alternatives such as WorkBuddy, Qoder and ZCode, Daoma works best in sync with me — probably just from long‑term habit.

My current workflow works like this: only one agent is active at any given time. The active agent holds the highest decision‑making authority apart from me, while the rest stay silent. Before the active agent runs out of quota — I only pay for DeepSeek; all other agent tools I use only free tiers, limited‑time offers or earned points🤣 — it leaves hand‑off documents. The next agent that spins up can pick up the work seamlessly. As long as you point them to a shared working directory and plan permission controls ahead of time, you can run this smooth hand‑off relay between agents.

Collapse
 
ai_pal profile image
The AI Pal •

There is a lot to unpack here, in a good way. An 8 GB VRAM setup running llama.cpp is very close to what I want to explore in the follow-up article.

Ollama makes the first setup pleasantly simple, but I can understand moving to llama.cpp once you want tighter control over memory use and model parameters. I am adding the Gemma quant you mentioned to my shortlist.

I also like your hand-off document approach. It sounds a bit like a shift change: one agent leaves clear notes, and the next one continues without starting from zero. If you are happy to share them, your context-size and GPU-offloading settings would be useful reference points for my testing.

Collapse
 
xulingfeng profile image
xulingfeng •

Happy to share — these are live values from my current setup, not guesses.

Model & quant: gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf (QAT, Q4_K_XL) + mmproj-F16.gguf (vision) + MTP draft model. Running on llama.cpp llama-server (b10712).

Context size: -c 131072 (128K). It loads fine on 8 GB VRAM with flash attention. I tested 64K too — that's the safe sweet spot if you want more headroom; 128K works but eats most of the card.

GPU offloading: -ngl 999 — everything on GPU, no CPU offload. That's the "finer-grained control" advantage over Ollama for me: I can pin exactly what goes where.

The flags that matter for 8 GB:

-ngl 999 --flash-attn on -np 1
--spec-type draft-mtp --spec-draft-n-max 5   # MTP speculative decoding = big speedup
--jinja                                        # proper chat templates
Enter fullscreen mode Exit fullscreen mode

Vision (--mmproj) and the MTP draft both live in the same process, so it's one memory footprint.

Real numbers from my card (RTX 4070S, 8 GB): model fully loaded sits at ~6.4 GB used, leaving ~1.7 GB free after the model + KV cache. Generation is roughly 90–150 t/s. If I need more headroom I'd drop context to 64K or cut the MTP draft before touching the quant.

One tip from hours of tinkering: with a thinking model like this one, give the caller a generous max_tokens (≥2048), otherwise the reasoning chain eats the budget and you get empty replies. Learned that the hard way.

Hope that helps your follow-up testing. I'll be reading it — your Ollama-vs-llama.cpp angle matches exactly how I ended up here.

Picked as gem