The most common interactions most of us will have with AI right now is through a cloud-based tool such as ChatGPT or Copilot. Those tools require a connection to the internet to use, but the trade-off there is that they're usable on basically any device.
But not everyone wants to rely on the cloud for their AI use, especially developers. For that, you want the LLM to be running locally on your machine. Which is where Ollama comes in.
Ollama is an inferencing tool that allows you to run a wide range of LLMs natively on your PC. It's not the only way you can achieve this, but it is one of the simplest.
Once up and running, there's a lot you can do with Ollama and the LLMs you're using through it, but the first stage is getting set up. So let's walk you through it.
What you need to run Ollama
Running Ollama itself isn't much of a drag and can be done on a wide range of hardware. It's compatible with Windows 11, macOS, and Linux, and you can even use it through your Linux distros inside Windows 11 via WSL.
Where you'll need some beefier hardware is actually running the LLMs. The bigger the model, the more horsepower you need. The models will need a GPU to run, right now, it hasn't been optimized to run on an NPU in a new Copilot+ PC.
But there are small models you can use on modest hardware. Google's Gemma 3, for example, has a 1 billion parameter model that requires only 2.3GB of VRAM to run the full model. This jumps to over 9GB of VRAM if you step up to the 4 billion parameter model.
It's a similar story with Meta's Llama 3.2, with the 1 billion parameter model only requiring a 4GB VRAM GPU to successfully run. This then jumps to 8GB for the 3 billion parameter model.
Essentially, if you have a fairly modern PC with at least 8GB of RAM and a dedicated GPU, you should be able to get some mileage using Ollama.
How to install Ollama on Windows 11
Installing Ollama on Windows 11 is as simple as downloading the installer from the website (or GitHub repo) and installing it.
That's literally all there is to it.
Once installed and subsequently opened, you won't see anything on your desktop. It runs entirely in the background, but you'll see its icon in your taskbar.
You can also check it's running properly by navigating to localhost:11434 in your web browser.
Installing and running your first LLM on Ollama
Ollama only has a CLI (command line interface) by default, so you'll need to fire up PowerShell (or WSL if that's where you installed it).
You can use a GUI with Ollama, but that's a different topic for a different day. It's worth being comfortable with the CLI, anyway.
Two main commands you need to know are as follows:
ollama pull <llm name>
ollama run <llm name> If you ask Ollama to run an LLM you don't currently have installed, it's smart enough to go and grab it first, then run. All you need to know is the correct name for the LLM you want to install, and those are readily available on the Ollama website.
For example, if you wanted to install the 1 billion parameter Google Gemma 3 LLM, you would enter the following:
ollama pull gemma3:1b By adding :1b after the name, you're specifying that you want the 1 billion parameter model. If you wanted the 4 billion parameter model, you would change it to :4b.
By running the models in your terminal it'll open up a familiar feeling chatbot experience allowing you to type in your prompts and get your responses. All running locally on your machine.
To leave the model and go back to PowerShell, you simply need to type /bye and it'll exit.
That covers the absolute basics of getting Ollama set up on your PC to use the host of LLMs available for it. The core experience is extremely simple and user-friendly, and requires almost no technical knowledge to set up. If I can do it, you can too!
You must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.
