Running AI Models Locally on Windows

Introduction

What if you could run your own AI assistant - one that works without the internet, costs nothing per message, never shares your data with anyone, and is available any time you need it? No subscriptions. No monthly limits. No worrying about what happens to your conversations.

That is exactly what running AI models locally means. Thanks to a free tool called Ollama, you can download and run powerful AI language models directly on your Windows laptop or desktop - the same way you would install any other application.

This guide will walk you through everything from scratch. No prior experience required. By the end, you will have your own AI assistant running on your machine, accessible both from the terminal and from a full browser-based chat interface that looks and feels just like ChatGPT.

Why Would You Want to Do This?

If ChatGPT and Claude already exist, why bother running something locally? Here are the real reasons people do it.

1. Complete Privacy
When you type into ChatGPT or Claude, your messages are sent to servers owned by OpenAI or Anthropic. Those companies may use your conversations to improve their models, and your data lives on their infrastructure. With a local model, your prompts never leave your computer. Not one word. This matters enormously when dealing with sensitive work documents, client data, medical information, or anything confidential.

2. Zero Cost
ChatGPT Plus costs around $20 per month. Claude Pro is similar. API usage costs money per request. Local models cost absolutely nothing after the initial download. You can run millions of messages and the bill stays at zero.

3. Works Completely Offline
Flying with no Wi-Fi? In a location with poor internet? Your local model does not care. Once downloaded, it runs entirely from your machine. No network required.

4. No Rate Limits or Message Caps
Cloud AI services throttle heavy users. You hit a limit, you wait. Local models have no such restriction. Run them as hard as you want for as long as you want.

5. Always Available
No service outages, no maintenance windows, no "ChatGPT is at capacity right now." Your model is there whenever you open your terminal.

6. Learning and Experimentation
Developers, researchers, and curious professionals use local models to experiment with AI, test prompts, build applications, and understand how these systems work - all without racking up API costs.

What You Need Before Starting

You do not need a powerful gaming machine. Here is what matters:

• Operating System: Windows 10 or Windows 11
• RAM: 8 GB minimum, 16 GB recommended
• Free Disk Space: At least 5 GB, ideally 15 GB or more
• GPU: Not required - models run on your CPU
• Internet: Only needed for the initial download

RAM is the most important factor. The more you have, the better the model you can run:

• 8 GB RAM - run 1B to 2B models. Fast but basic.
• 16 GB RAM - run 2B to 4B models. Good balance of speed and quality.
• 32 GB RAM - run 7B to 13B models. Noticeably more capable.
• 48 GB RAM or more - run 70B models. Closest to GPT-4 quality locally.

Understanding Model Sizes

When you see terms like "2B" or "7B", the B stands for billion parameters. Parameters are the internal numbers that make up the model's knowledge and reasoning ability. More parameters generally means more capable responses, larger file size, more RAM required, and slower responses on CPU.

Think of it like engine size in a car. A 2B model is a small efficient engine - fast and economical. A 70B model is a high-performance engine - far more powerful but needs much more fuel (RAM) and takes longer to get going. For most everyday tasks on a standard laptop, a 3B to 7B model hits the sweet spot perfectly.

The Models Available on Ollama and What They Are Best At

Models come in different sizes. The bigger the parameter count, the more capable but the more RAM and time required. Here is a full breakdown from smallest to largest.

Small Models - 1B to 4B Parameters

Best for: laptops with 8–16 GB RAM, everyday tasks, fast responses

gemma2:2b - Google Gemma 2 (2 Billion)
Size: ~1.6 GB | RAM needed: 4–6 GB

ollama pull gemma2:2b

Google's lightweight model. The fastest option on CPU. Ideal for quick questions, short summaries, and simple drafts. Not suited for complex reasoning but excellent when speed matters more than depth.

Best used for:

• Quick factual questions
• Summarising short pieces of text
• Simple email drafting
• When you need a fast answer and don't need deep reasoning

phi3:mini - Microsoft Phi-3 Mini (3.8 Billion)
Size: ~2.2 GB | RAM needed: 6–8 GB

ollama pull phi3:mini

Trained on high-quality textbook-style content. Punches well above its weight on structured tasks. Best small model for code, JSON output, and step-by-step reasoning. If you are a developer this should be your first download.

Best used for:

• Coding questions and debugging
• Structured outputs like JSON or formatted data
• Step-by-step explanations
• Maths and logic problems
• Following detailed instructions precisely

llama3.2:3b - Meta Llama 3.2 (3 Billion)
Size: ~2.0 GB | RAM needed: 6–8 GB

ollama pull llama3.2:3b

Meta's latest small model and the best all-rounder in the 3B range. Handles a wide variety of tasks reliably. This is the model most people should start with.

Best used for:

• General conversation and everyday questions
• Writing assistance - emails, reports, summaries
• Explaining complex topics in plain language
• Brainstorming and idea generation
• Basic coding help
• Translation and language tasks

qwen2.5:3b - Alibaba Qwen 2.5 (3 Billion)
Size: ~1.9 GB | RAM needed: 6–8 GB

ollama pull qwen2.5:3b

Alibaba's small model with exceptional multilingual capability and reliable structured output. If you work in multiple languages or need consistent JSON and formatted data, this is your pick at the small end.

Best used for:

• Structured data extraction
• JSON and formatted output tasks
• Multilingual tasks - works well in multiple languages including non-English ones
• Data classification and categorisation
• Anything where consistent output format matters

mistral:7b - Mistral AI (7 Billion)
Size: ~4.1 GB | RAM needed: 8–12 GB

ollama pull mistral:7b

One of the most popular open source models ever released. Mistral 7B outperforms many models twice its size. Excellent all-rounder that handles nuanced writing, reasoning, and coding far better than the 3B models. If your machine has 16 GB RAM, start here instead of the smaller models. The quality jump from 3B to 7B is very noticeable.

Best used for:

• Everything llama3.2:3b does, but better
• Nuanced writing with more depth and flow
• Coding with fewer errors
• Longer documents where context matters
• When you want noticeably better quality and have the RAM

Medium Models - 7B to 13B Parameters

Best for: machines with 16 GB RAM, more complex tasks, better reasoning

llama3.1:8b - Meta Llama 3.1 (8 Billion)
Size: ~4.7 GB | RAM needed: 10–14 GB

ollama pull llama3.1:8b

One of Meta's strongest open source releases. At 8B parameters it handles complex instructions, long documents, nuanced writing, and multi-step reasoning much better than the 3B version. This is the sweet spot for most people with 16 GB RAM - capable enough for serious work, still manageable on CPU.

Best used for:

• Complex multi-step tasks and analysis
• Long document summarisation
• Detailed writing with nuance
• Research assistance
• Technical explanations

gemma2:9b - Google Gemma 2 (9 Billion)
Size: ~5.4 GB | RAM needed: 12–16 GB

ollama pull gemma2:9b

The bigger sibling of gemma2:2b. Substantially more capable while still being efficient. Strong at instruction following, summarisation of long texts, and factual question answering. Google specifically optimised this size for local deployment.

Best used for:

• Summarising long reports and documents
• Factual question answering with better accuracy
• Instruction-following tasks requiring precision
• When you want Google-quality output locally

deepseek-coder:6.7b - DeepSeek Coder (6.7 Billion)
Size: ~3.8 GB | RAM needed: 8–10 GB

ollama pull deepseek-coder:6.7b

A coding-specialised model that performs remarkably well for its size. Strong at Python, JavaScript, and SQL in particular. A practical alternative to CodeLlama if you want solid code assistance without large RAM requirements.

Best used for:

• Python, JavaScript, SQL code generation
• Debugging and error fixing
• Code explanation and documentation
• Writing unit tests

codellama:13b - Meta Code Llama (13 Billion)
Size: ~7.4 GB | RAM needed: 14–18 GB

ollama pull codellama:13b

Built specifically for code. If you are a developer and code assistance is your primary use case, CodeLlama at 13B is one of the best local options available. Handles Python, JavaScript, SQL, and many other languages. Can explain existing code, generate new code, and debug errors. Not designed for general conversation.

Best used for:

• Serious code generation across many languages
• Explaining and refactoring existing codebases
• Generating tests and documentation
• Complex debugging

phi3:medium - Microsoft Phi-3 Medium (14 Billion)
Size: ~7.9 GB | RAM needed: 14–18 GB

ollama pull phi3:medium

Microsoft's larger Phi-3 model. Takes everything phi3:mini does well - structured reasoning, code, precise instructions - and does it significantly better. Recommended for developers who need reliable code generation or anyone doing complex data tasks.

Best used for:

• Advanced coding and structured output
• Complex step-by-step reasoning
• Data tasks requiring high precision
• When phi3:mini is good but not quite good enough

mistral-nemo:12b - Mistral AI (12 Billion)
Size: ~7.1 GB | RAM needed: 14–18 GB

ollama pull mistral-nemo:12b

Mistral's newer medium model built in collaboration with NVIDIA. Very strong at instruction following, long context understanding, and technical writing. A step up from mistral:7b in quality while remaining practical on a 16 GB machine.

Best used for:

• Long context tasks where the model needs to remember a lot
• Technical documentation and writing
• Detailed instruction following
• Professional report drafting

llava:13b - LLaVA Multimodal (13 Billion)
Size: ~8.0 GB | RAM needed: 14–18 GB

ollama pull llava:13b

One of the very few local models that can look at images. Send it a photo, screenshot, diagram, or chart and ask questions about it. Not as capable as GPT-4 Vision but genuinely useful and completely private. If you need local image understanding, this is your only realistic option at this size.

Best used for:

• Asking questions about screenshots or photos
• Describing diagrams and charts
• Reading text from images
• Any task where you need to share a visual

Large Models - 27B to 70B Parameters

Best for: machines with 32 GB RAM or more, highest quality output, closest to cloud AI

gemma2:27b - Google Gemma 2 (27 Billion)
Size: ~16 GB | RAM needed: 20–28 GB

ollama pull gemma2:27b

Google's largest Gemma model. Strong across writing, reasoning, and instruction following. Well optimised for local inference compared to other models of similar size. A good first step into large models for anyone with 32 GB RAM.

Best used for:

• High-quality long-form writing
• Complex analysis and research assistance
• Nuanced reasoning that smaller models struggle with
• Professional-grade document drafting

deepseek-r1:32b - DeepSeek (32 Billion)
Size: ~19 GB | RAM needed: 24–32 GB

ollama pull deepseek-r1:32b

DeepSeek's reasoning-focused model that caused a stir when released for matching or beating much larger Western models on reasoning benchmarks. Exceptional at mathematical reasoning, logic problems, and complex analytical thinking.

Best used for:

• Mathematical reasoning and problem solving
• Logic and analytical tasks
• Data interpretation
• Step-by-step complex reasoning chains
• Technical problem solving

codellama:34b - Meta Code Llama (34 Billion)
Size: ~20 GB | RAM needed: 24–32 GB

ollama pull codellama:34b

The large version of CodeLlama. For serious developers who need the highest quality local code assistance available. Handles complex codebases, generates tests, writes documentation, and debugs across many languages with significantly higher accuracy than the 13B version.

Best used for:

• Enterprise-grade code generation
• Complex multi-file code understanding
• Architecture-level coding decisions
• When codellama:13b is not good enough

mixtral:8x7b - Mistral AI Mixtral (47 Billion effective)
Size: ~26 GB | RAM needed: 32–48 GB

ollama pull mixtral:8x7b

A clever architecture called Mixture of Experts - it has 47 billion total parameters but only activates around 13 billion at a time, making it faster than a full 47B model. Excellent at complex reasoning, coding, and multilingual tasks. Delivers near GPT-4-level quality for many tasks.

Best used for:

• Complex reasoning and analysis
• High-quality multilingual work
• Advanced coding tasks
• When you want GPT-4-level quality locally and have the RAM

llama3.1:70b - Meta Llama 3.1 (70 Billion)
Size: ~40 GB | RAM needed: 48 GB or more

ollama pull llama3.1:70b

One of the most capable open source models in existence. Competitive with GPT-4 on many benchmarks. Handles complex legal analysis, long-form writing, advanced reasoning, and nuanced understanding at a level the smaller models simply cannot match. Not realistic on a standard laptop - best run on a high-RAM workstation or a machine with a dedicated GPU.

Best used for:

• Tasks that currently require cloud AI quality
• Complex legal, financial, or medical document analysis
• High-quality long-form content creation
• Advanced multi-step reasoning
• Anything where you need the very best local model available

Specialised Models

nomic-embed-text - Nomic (Embedding Model)
Size: ~274 MB | RAM needed: Minimal

ollama pull nomic-embed-text

Not a chat model - an embedding model. Used by developers to convert text into numerical vectors for semantic search, document similarity, and RAG pipelines. Tiny and fast. Not for conversation but essential for many AI application development workflows.

Best used for:

• Building semantic search systems
• Document similarity comparison
• RAG pipeline development
• AI application development

Full Model Comparison Table

Quick Reference - Which Model for Which Task

What Should You Run Given Your Machine?

8 GB RAM:
Stick to gemma2:2b, llama3.2:3b, or phi3:mini. These run comfortably and handle most everyday tasks well.

16 GB RAM (most laptops):
Start with llama3.2:3b or mistral:7b. Add phi3:mini for code tasks and deepseek-coder:6.7b if you write a lot of code. You can also push to llama3.1:8b if you close other applications while running it.

32 GB RAM:
You can run llama3.1:8b, deepseek-r1:32b, or gemma2:27b comfortably. This is where local models start feeling genuinely close to cloud AI quality. The difference from 16 GB models is very noticeable.

32 GB RAM + GPU:
Ollama automatically uses your GPU if one is present. Models that took 30 seconds on CPU now respond in 2 to 3 seconds. At this point local AI becomes a serious daily driver that rivals cloud services on speed.

48 GB RAM or more:
You can run llama3.1:70b - one of the best open source models available anywhere. At this level you are genuinely competitive with GPT-4 for many tasks, running completely privately and at zero cost per message.

Advantages Over ChatGPT, Claude, and Other Cloud AI

Disadvantages - Being Honest

Local models are impressive but they are not a complete replacement for cloud AI in every situation. Here is what you should know:

They are slower. Cloud AI runs on massive GPU clusters. Your CPU takes longer to generate responses. Expect 5 to 30 seconds per paragraph depending on model size. Fine for most tasks, but slower than ChatGPT's near-instant replies.

Smaller models make more mistakes. The 2B to 4B models sometimes confidently state incorrect information. The larger the model the more reliable it becomes, but always verify important facts before acting on them.

No real-time information. These models were trained on data up to a certain date and cannot browse the internet. They cannot tell you today's news, current prices, or recent events.

Large models need large hardware. The most capable models require 32 GB or 48 GB RAM. These are not realistic on a standard laptop. If you only have 16 GB you are limited to the small and medium range.

First run is slow. The first time you run a model after starting Ollama, it loads from disk into RAM. This can take 10 to 30 seconds. After that, responses are faster.

Step 1 - Install Ollama on Windows

Open your browser and go to https://ollama.com/download and click Download for Windows.

Run the installer called OllamaSetup.exe. It installs quietly and starts a local server on your machine at http://localhost:11434. You will see the Ollama icon appear in your system tray.

Verify it installed correctly by opening PowerShell and running:

ollama --version

You should see a version number printed. If you see an error, close and reopen PowerShell and try again.

Step 2 - Download Your First Model

In PowerShell, run this to download the recommended all-rounder:

ollama pull llama3.2:3b

Then download others as needed. Each is around 2 to 8 GB:

ollama pull mistral:7b
ollama pull phi3:mini
ollama pull gemma2:2b
ollama pull qwen2.5:3b

See all downloaded models on your machine with:

ollama list

Step 3 - Chat via the Terminal

Start a conversation directly in PowerShell:

ollama run llama3.2:3b

You will see a >>> prompt. Just type your question and press Enter:

>>> What is the difference between RAM and storage?

The model responds. Ask follow-up questions just like ChatGPT. To exit:

>>> /bye

Try the other models the same way:

ollama run mistral:7b
ollama run phi3:mini
ollama run gemma2:2b

Useful terminal commands:

Step 4 - Use a Proper Chat Interface in the Browser

The terminal works but a full browser-based interface is much more comfortable for everyday use. Open WebUI gives you a complete ChatGPT-style interface - model selector, conversation history, dark mode, everything - running entirely on your machine.

Install Open WebUI:

pip install open-webui

Start it:

open-webui serve

Leave this terminal window open. Then open your browser and go to:

http://localhost:8080

On your first visit you will be asked to create a local account. Use any username, email, and password you like - this account is stored entirely on your machine, nothing goes anywhere.

Once logged in you will see a full chat interface. Click the model dropdown at the top and select whichever model you want, then start chatting normally.

To stop Open WebUI, press Ctrl + C in the terminal window running it. To start it again next time, run open-webui serve and visit http://localhost:8080.

Tips for Getting Better Responses

Be specific about what you want.
Instead of "help me with this email" try "rewrite this email in a professional tone, keep it under 100 words, and make the call to action clear."

Tell it the format you want.
"Explain this in bullet points." "Give me a table comparing these options." "Respond in plain language for a non-technical audience."

Give it context.
"I am a software engineer working on a Python API. Explain this error message and suggest a fix."

Ask it to think step by step.
Simply adding "walk me through this step by step" significantly improves reasoning quality on smaller models.

Switch models for different tasks.
Use gemma2:2b for quick questions, phi3:mini or deepseek-coder for code, llama3.2:3b or mistral:7b for general writing, qwen2.5:3b for structured output, deepseek-r1:32b for analytical reasoning if your machine can handle it.

Upgrade your model when results disappoint.
If a response is not good enough, try the next size up. The quality difference between a 3B and 8B model on the same task is often significant.

Frequently Asked Questions

Does it use the internet after the models are downloaded?
No. Everything runs completely offline. Nothing you type leaves your machine.

Will it slow down my computer?
During inference your CPU usage will spike for a few seconds while generating a response. Other tasks may feel slightly sluggish during that time. Afterwards it returns to normal.

Can I use it for sensitive work documents?
Yes. Nothing leaves your machine, making it suitable for confidential documents, client data, and internal company information.

What if I want a more capable model than the ones listed?
Browse the full library at https://ollama.com/library. New models are added regularly as the open source AI community releases them.

How do I update a model when a newer version comes out?
Just pull it again: ollama pull llama3.2:3b. Ollama downloads the updated version automatically.

Can I run multiple models at the same time?
Yes, but each model loaded into RAM takes up its full allocation. On 16 GB RAM, running two 3B models simultaneously would use most of your available memory.

What if I have a GPU?
Ollama detects your GPU automatically and uses it. You do not need to configure anything. Response times drop dramatically - from 15 to 30 seconds down to 1 to 3 seconds on a mid-range GPU.

Summary

Running AI models locally on Windows is simpler than most people expect. Install one tool, download a model, and you are ready to go. The whole setup takes under 30 minutes.

You now have a private, free, offline AI assistant that runs on hardware you already own. It will not replace cloud AI for every task - complex reasoning and real-time information are still better handled by GPT-4 or Claude - but for the majority of everyday questions, writing assistance, coding help, and data tasks, your local model will serve you very well.

Start with llama3.2:3b for general use. Move to mistral:7b when you want better quality. Add phi3:mini for code. Explore the larger models as your needs grow. And enjoy having your own AI that answers to nobody but you.

Browse the full model library at https://ollama.com/library