Configuring a Private Local AI Coding Agent with Ollama on Windows 11

Recently, I wanted to experiment with running my own AI coding assistant locally instead of relying entirely on cloud-based AI services.

The motivation was simple.

For some day-to-day development tasks, I don't necessarily need a large cloud model. Sometimes I just need a quick lookup, a small code suggestion, an explanation, or help understanding a piece of code.

Using a paid AI service for every small task can become expensive, especially when usage limits or credits are involved.

There was another important consideration as well: keeping development-related information on my own machine whenever possible.

So I started looking into local AI models and came across Ollama.

Fortunately, my laptop had enough resources to experiment with smaller coding models.

In this article, I'll walk through the setup I used on Windows 11, the models I tested, and how I connected Ollama with VS Code.

My Laptop Configuration

This is the machine I used for the experiment:

CPU: Intel Core i7, 1.30 GHz
RAM: 32 GB Kingston
GPU: NVIDIA GeForce MX250 / Intel Iris Plus Graphics
Storage: 500 GB SSD
OS: Windows 11

This isn't a high-end AI workstation, so I wasn't expecting to run large models efficiently.

My goal was simply to find a model small enough to run reasonably well while still being useful for software development.

Installing Ollama

There are several ways to install Ollama on Windows.

Although an installer is available, I prefer keeping development tools organized manually, so I downloaded the Windows ZIP distribution from the Ollama releases page.

I extracted Ollama into:

C:\tools\ai\ollama\bin\

I also created a separate directory for the downloaded models:

C:\tools\ai\ollama\models\

Keeping the models in a location that I control makes it easier to understand where the potentially large model files are being stored.

Selecting a Coding Model

Ollama provides a library of models that can be downloaded and run locally.

For this experiment, I selected qwen2.5-coder:

qwen2.5-coder

and tested two variants:

qwen2.5-coder:1.5b
qwen2.5-coder:7b

The 1.5b and 7b refer to the approximate number of parameters in the models.

I was particularly interested in seeing how these two sizes would behave on my relatively modest hardware.

Configuring the Ollama Model Directory

One of the first things I wanted to control was where Ollama stores its models.

I created a start.bat file inside:

C:\tools\ai\ollama\bin\start.bat

with the following:

SET OLLAMA_MODELS=C:\tools\ai\ollama\models

ollama serve

The OLLAMA_MODELS environment variable tells Ollama where I wanted the model files to be stored.

This is particularly useful when experimenting with multiple models because model files can consume several gigabytes of disk space.

Starting the Ollama Server

I opened Windows Terminal with administrator privileges and navigated to the Ollama directory:

PS C:\> cd .\tools\ai\ollama\bin

PS C:\tools\ai\ollama\bin> .\start.bat

Ollama starts a local server, normally available at:

http://localhost:11434

Opening that address in a browser showed:

Ollama is running

At this point, the local Ollama server was ready.

Downloading and Running the Models

I opened another terminal window and ran:

PS C:\> cd .\tools\ai\ollama\bin

PS C:\tools\ai\ollama\bin> .\ollama.exe run qwen2.5-coder:1.5b

I also tested:

PS C:\tools\ai\ollama\bin> .\ollama.exe run qwen2.5-coder:7b

The run command downloads the model if it isn't already available and then starts it.

You can also download a model separately using:

ollama pull qwen2.5-coder:1.5b
ollama pull qwen2.5-coder:7b

I preferred using run during my initial testing because I could see the download progress and immediately start interacting with the model afterward.

How Large Were the Models?

After downloading the models, I checked the local model list:

PS C:\tools\ai\ollama\bin> .\ollama.exe list

My output looked like this:

NAME                  ID             SIZE      MODIFIED

qwen2.5-coder:1.5b    d7372fd82851   986 MB    34 hours ago
qwen2.5-coder:7b      dae161e27b0e   4.7 GB    34 hours ago

This was one of the interesting parts of the experiment.

The model name might suggest that a 7B model should require 7 GB of storage, but the actual size depends on factors such as the model format and quantization.

So it is better to check the actual model size rather than estimating storage requirements purely from the parameter count.

My Experience with the 1.5B and 7B Models

This was probably the most useful part of the experiment for me.

The qwen2.5-coder:1.5b model was reasonably responsive on my laptop.

For simple coding questions, explanations, and small pieces of code, the response speed was good enough for me to actually use it.

Then I tried:

qwen2.5-coder:7b

The difference was noticeable.

The 7B model was considerably slower on my hardware, particularly when I wanted to use it interactively for coding.

The model itself wasn't necessarily the problem. My hardware simply wasn't an ideal environment for running a larger model efficiently.

This was an important reminder for me:

A larger model isn't automatically a better experience if your hardware can't run it comfortably.

For my particular laptop, the smaller model provided a much more practical interactive experience.

Running Ollama as a Windows Service

After getting the basic setup working, I wanted Ollama to run as a Windows service instead of manually starting the server every time.

For this, I used NSSM — Non-Sucking Service Manager.

I downloaded NSSM and configured Ollama as a Windows service.

The basic configuration was:

Application

Set the Ollama executable as the application:

ollama.exe

Arguments

Use:

serve

Startup Directory

Set the startup directory to the directory containing ollama.exe.

I also configured the service name as:

Ollama

After saving the configuration, I could start and manage Ollama through the Windows Services application.

If I needed to modify the configuration later, NSSM also provides:

nssm edit Ollama

This approach meant I didn't have to manually launch ollama serve every time I wanted to use my local AI environment.

Connecting Ollama with VS Code

Once Ollama was working independently, the next step was making it useful inside my development environment.

I use VS Code for development, so I experimented with several AI coding extensions:

For this particular setup, I found Continue interesting because it provides several different ways to interact with models rather than focusing only on autonomous coding.

For example, it supports workflows such as:

Chat
Edit
Agent
Autocomplete
Codebase context
Configurable models
Different models for different tasks
Local/Ollama models
Custom prompts and rules

It also fits well with the technologies I commonly work with, including:

Using Different Models for Different Tasks

One of the ideas I particularly liked was not treating every AI task the same way.

A smaller model can potentially be useful for lightweight tasks such as autocomplete, while a larger model can be used for more complex interactions when the hardware can handle it.

For example, my Continue configuration looked conceptually like this:

                    Ollama
                       │
              ┌────────┴────────┐
              │                 │
       Qwen2.5-Coder 7B    Qwen2.5-Coder 1.5B
              │                 │
        ┌─────┼─────┐           │
        │     │     │           │
       Chat  Edit  Agent   Autocomplete

My configuration used the 7B model for more involved interactions and the smaller model for autocomplete:

name: My Local Config
version: 0.0.1
schema: v1

models:
  - name: Qwen2.5-Coder 7B
    provider: ollama
    model: qwen2.5-coder:7b
    roles:
      - chat
      - edit
      - apply

  - name: Qwen2.5-Coder 1.5B
    provider: ollama
    model: qwen2.5-coder:1.5b
    roles:
      - autocomplete

This approach is interesting because it allows the development workflow to be designed around the capabilities of the available hardware.

Instead of asking one model to handle everything, different models can potentially serve different purposes.

Hardware Matters More Than I Initially Expected

Running local AI changed the way I think about hardware requirements.

With cloud-based AI, most of the heavy computation happens on remote infrastructure.

With a local model, your own machine has to perform the inference.

That means several hardware components can influence the experience:

CPU performance
Available RAM
GPU capabilities and VRAM
Storage speed
Model size
Model quantization
Context length
Number of concurrent workloads

In my case, the 32 GB of RAM and SSD were useful, but the relatively modest GPU meant that larger models were not particularly comfortable to use.

So before downloading several large models, it is worth checking whether your hardware can actually run them at a useful speed.

How Much Disk Space Do You Need?

This depends heavily on the models you choose.

A small coding model may only require around 1 GB, while larger models can consume several gigabytes each.

For example, in my experiment:

qwen2.5-coder:1.5b  → ~986 MB
qwen2.5-coder:7b    → ~4.7 GB

If you are only experimenting with one or two relatively small models, you don't necessarily need a huge amount of storage.

However, model collections can grow quickly.

It is therefore a good idea to decide where you want your local AI models stored before downloading a large number of them.

Is Local AI Worth Trying?

For me, the biggest benefit of this experiment wasn't replacing cloud AI completely.

It was having another option.

There are plenty of situations where a cloud-based model will still be more capable, particularly when working with complex reasoning, large codebases, or tasks that require capabilities beyond what a small local model can provide.

But there are also many small development tasks where having a local model available is convenient.

For example:

"Explain this function."

"Generate a simple TypeScript interface."

"Give me a regex for this pattern."

"Convert this JavaScript function to TypeScript."

"What does this error mean?"

"Generate a basic unit test."

"Complete this small piece of code."

For these types of tasks, having a local model available can be useful without sending every small interaction to a remote service.

Final Thoughts

This experiment gave me a better understanding of what it actually takes to run AI locally.

The most important lesson for me was that model size, hardware, and actual workload need to be considered together.

A 7B model may sound much more attractive than a 1.5B model, but if the larger model is too slow on your machine, it may not provide a better day-to-day development experience.

In my case, the qwen2.5-coder:1.5b model was fast enough to be practical, while the 7b model was considerably slower for interactive coding.

The interesting part is that this setup doesn't require a dedicated AI server.

A reasonably capable development machine can be enough to start experimenting with local AI.

And once Ollama is running, it opens the door to experimenting with different models, IDE integrations, coding assistants, and completely local AI workflows.

For me, this is less about replacing every cloud AI service and more about having another tool in the developer toolbox.

That's probably the most useful way to look at local AI today.