Ollama AI

Ollama is a tool for running open-weight large language models locally — on a laptop or workstation GPU/CPU — packaging model download, quantization variants, and a local inference server behind a small CLI and REST API.

CLI basics

ollama pull llama3.2          # download a model from Ollama's library
ollama run llama3.2           # interactive chat in the terminal
ollama list                   # show locally downloaded models
ollama ps                     # show currently loaded/running models
ollama rm llama3.2            # remove a downloaded model

ollama run <model> both pulls (if not already present) and starts an interactive session in one command — the fastest path to trying a model.

Local API

Ollama runs a local server (default http://localhost:11434) with its own REST API, and separately exposes an OpenAI-compatible endpoint so existing OpenAI-client code can point at it with minimal changes.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# Native Ollama Python client
import ollama
resp = ollama.chat(model="llama3.2", messages=[{"role": "user", "content": "Hi"}])
print(resp["message"]["content"])

# OpenAI-compatible client, pointed at Ollama instead of OpenAI
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is unchecked, any string works
resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hi"}],
)

Modelfile — customizing a model

A Modelfile layers a system prompt, parameters, or a different base weight file on top of an existing model, similar in spirit to a Dockerfile:

FROM llama3.2
PARAMETER temperature 0.3
SYSTEM """
You are a terse code-review assistant. Answer in bullet points only.
"""

ollama create my-reviewer -f ./Modelfile
ollama run my-reviewer

Common pitfalls

• Picking a model too large for the available VRAM/RAM — Ollama will still try to run it, often by spilling to CPU/swap, which can be drastically slower than a smaller or more aggressively quantized variant of the same model. Check a model's size/quantization tags on the library page against available memory first.
• Assuming the OpenAI-compatible endpoint supports every OpenAI feature — it covers the common chat/completions/streaming/tool-calling shape, not every parameter or the newer Responses API; check Ollama's docs for current coverage before porting complex OpenAI-specific code.
• Forgetting the server needs to be running — the CLI auto-starts it on most installs, but in a container or minimal environment ollama serve may need to be started explicitly before ollama run/API calls work.
• Not distinguishing a model's Ollama library tag from its upstream name/license — the library repackages open-weight models from various sources; the license terms are the upstream model's, not Ollama's.

Learn more

• Ollama docs
• Ollama model library — browse available models and quantization tags.
• Ollama API reference
• Ollama OpenAI compatibility docs