Ollama Tutorial: Run Local LLMs From the Command Line

What is Ollama?

Ollama is an open-source tool for running open-weight large language models on your own hardware. Instead of sending prompts to a hosted API, you pull a model once and run inference entirely on your laptop or server. Ollama handles the awkward parts of local inference - downloading and caching weights, picking a sensible quantization, applying the correct chat template, and managing GPU or CPU memory - behind a single command and a small HTTP API.

The appeal is control. Your prompts and data never leave the machine, there are no per-token costs, and you can work offline. That makes Ollama a natural fit for prototyping, privacy-sensitive workloads, CI pipelines that need a local model, and learning how LLMs behave without paying per request. It supports a large catalog of models - Llama, Qwen, Gemma, Mistral, Phi, embedding models, and more - all addressable by short names like llama3.2 or qwen3:8b.

How Ollama works (key concepts)

Ollama has two halves: a background server and a client. The server (ollama serve) is a long-running process that loads models into memory and answers HTTP requests on http://localhost:11434. The CLI and the libraries are just clients of that API. On macOS and Windows the desktop app starts the server for you; on Linux the install script registers a systemd service.

A model in Ollama is more than weights. Each model is a bundle of the weight file, the prompt template that formats messages, default parameters such as temperature and context length, and an optional system prompt. Models are tagged like Docker images: llama3.2:1b and llama3.2:3b are different sizes of the same family, and latest is the default tag.

Most models are distributed quantized, meaning the weights are compressed to fewer bits (commonly 4-bit, shown as suffixes like q4_K_M) to cut memory use and speed up inference, with a small accuracy cost. A 4-bit 8B model needs roughly 5-6 GB of memory, which is why it runs on ordinary hardware.

Ollama keeps a model resident for a few minutes after a request (controlled by keep_alive) so back-to-back calls skip the load time. The first call to a cold model pays a one-time load cost; later calls are fast.

Installing Ollama

On Linux, the official script installs the binary and sets up the systemd service:

curl -fsSL https://ollama.com/install.sh | sh

On macOS, install the app from the download page or use Homebrew:

brew install ollama

Verify the install and pull your first model. ollama run downloads the model if it is not present, then drops you into an interactive prompt:

ollama --version

# Pull a small, fast model and chat with it
ollama run llama3.2:1b

# Other day-to-day commands
ollama list          # show installed models
ollama ps            # show models currently loaded in memory
ollama pull qwen3:8b # download without running
ollama rm llama3.2:1b

If you installed manually or want to run the server in the foreground (useful on a headless box or in a container), start it yourself:

ollama serve

The server listens on 127.0.0.1:11434. To accept connections from other machines, set OLLAMA_HOST=0.0.0.0 before starting it - but treat that as exposing an unauthenticated inference endpoint and firewall it accordingly.

Calling the API and building a custom model

The CLI is convenient, but most applications use the HTTP API. Here is the native chat endpoint with streaming turned off so you get one JSON object back:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:1b",
  "messages": [
    { "role": "user", "content": "Explain a database index in one sentence." }
  ],
  "stream": false
}'

The official Python library wraps the same API. Install it with pip install ollama:

import ollama

response = ollama.chat(
    model="llama3.2:1b",
    messages=[
        {"role": "system", "content": "You are a terse senior engineer."},
        {"role": "user", "content": "When should I use a message queue?"},
    ],
)
print(response["message"]["content"])

# Streaming token by token
for chunk in ollama.chat(
    model="llama3.2:1b",
    messages=[{"role": "user", "content": "List three caching strategies."}],
    stream=True,
):
    print(chunk["message"]["content"], end="", flush=True)

You can also create your own model variant with a Modelfile - a small config that sets a base model, a system prompt, and parameters. Save this as Modelfile:

FROM llama3.2:1b
SYSTEM "You are a code reviewer. Reply only with bullet points."
PARAMETER temperature 0.2
PARAMETER num_ctx 4096

Then build and run it:

ollama create reviewer -f Modelfile
ollama run reviewer "Review: def add(a,b): return a-b"

The new reviewer model now uses your system prompt and parameters every time, and other clients can call it by name through the API.

When to use Ollama

Reach for Ollama when you want local, low-friction inference: prototyping agents and RAG pipelines without paying for a cloud API, keeping sensitive prompts on-premises, getting more repeatable outputs in tests (set a fixed seed and a temperature of 0), or simply learning how different open models behave. Its OpenAI-compatible endpoint at /v1/chat/completions means you can develop against a local model and later swap the base URL to a hosted provider with little code change.

It is a weaker fit when you need frontier-level reasoning quality, very large context windows, or high-throughput concurrent serving for many users - hosted APIs or dedicated inference servers like vLLM handle those better. Model quality is also bounded by your hardware: a 4-bit small model on a CPU will not match a large hosted model. Used within those limits, Ollama is one of the quickest ways to get a real LLM running on your own machine.