Ollama: the complete guide to running AI locally with privacy

What is Ollama

Ollama is an open-source tool that lets you run Large Language Models (LLMs) locally — on your own machine, with no internet, no API costs, and no sharing data with third parties. With a single command (ollama run llama3) you have a conversational AI model running in seconds, with the same quality as paid cloud services, but with total privacy [1].

Released in 2023, Ollama quickly became one of the most popular tools for running AI locally, with over 100 thousand stars on GitHub and an active community. It supports the main open-source models: Llama 3 (Meta), Gemma 2 (Google), Qwen 2.5 (Alibaba), Mistral, Phi-3 (Microsoft), among dozens of others.

Why run LLMs locally?

At a time when data privacy is increasingly critical — especially with the enforcement of the GDPR and the LGPD — the ability to process sensitive information without sending it to third-party servers is a competitive advantage. Here are the main reasons:

• Total privacy — your data never leaves the machine. Ideal for confidential documents, proprietary code, or client information.
• Zero cost per token — once installed, there are no API bills. The only cost is the hardware.
• Works offline — no dependency on an internet connection or cloud services that can go down.
• Minimal latency — with no round trip to the cloud, responses are instantaneous for most tasks.
• Personalization — you can create custom models (Modelfiles) with your own instructions, knowledge, and parameters.

Installation — simple on any system

Ollama is available for macOS, Linux, and Windows. Installation is one terminal line [2]:

macOS and Windows

Just download the installer from ollama.com/download and run it. There are no additional dependencies — Ollama includes everything needed.

Linux

curl -fsSL https://ollama.com/install.sh | sh

Docker (alternative)

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

In all cases, Ollama is left with a service running in the background (on port 11434) and a CLI client available in the terminal.

First steps

After installation, running a model is trivial:

ollama run llama3

On the first run, the model is downloaded (4.7GB for Llama 3 8B) and becomes available for immediate conversation. From there, you can interact in the terminal as in a chat:

>>> Explain the concept of Zero Trust in three sentences.
Zero Trust is a security model in which nothing and no one is trusted by default. Every access request is verified and authenticated, regardless of whether it comes from inside or outside the network. The core principle is "never trust, always verify".

Other essential commands:

• ollama pull mistral — download a model without running it
• ollama list — view installed models
• ollama rm llama3 — remove a model
• ollama show llama3 — view details (parameters, architecture, template)

The model catalog

Ollama has a growing library of models at ollama.com/library. The most popular include:

• Llama 3 (8B / 70B) — Meta's model, excellent for general use, reasoning, and code.
• Gemma 2 (9B / 27B) — Google's model, balanced between quality and size.
• Qwen 2.5 (7B / 72B) — from Alibaba, with excellent multilingual support (including Portuguese).
• Mistral (7B) — efficient and fast, ideal for modest hardware.
• Phi-3 (mini) — from Microsoft, ultra-lightweight (3.8B), runs on almost any machine.
• Code Llama / Qwen Coder — specialized in programming.
• LLaVA — multimodal: processes text AND images.

The choice depends on the available hardware and the use case. 7-8B models run comfortably on 8GB of RAM; 70B models require 32GB+ or aggressive quantization.

REST API — integration with applications

Ollama exposes a REST API at localhost:11434, compatible with OpenAI's format in many cases. This means you can replace the ChatGPT API with local Ollama with minimal changes to the code:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Write a business email in Portuguese.",
  "stream": false
}'

There are official SDKs for Python and JavaScript/TypeScript, plus integrations with frameworks like LangChain, LlamaIndex, and AutoGen.

Open WebUI — an interface like ChatGPT

For those who prefer a graphical interface over the terminal, Open WebUI (formerly Ollama WebUI) offers an experience identical to ChatGPT, but running locally on top of Ollama. Features include:

• Model selection from a menu
• Persistent conversation history
• Document upload (RAG)
• Conversation sharing
• Multi-user support

Installation via Docker: docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

Practical use cases

• Local programming assistant — edit code with AI suggestions without sending proprietary code to the cloud.
• Confidential document analysis — process contracts, financial reports, or client data with total privacy.
• Internal support chatbot — company knowledge base indexed locally.
• Task automation — integrate with scripts and CI/CD pipelines.
• AI prototyping — test ideas quickly without API costs before scaling to the cloud.
• Educational — learn about LLMs hands-on, with no financial barrier.

Performance and hardware requirements

Ollama uses quantization (4-bit by default) to reduce model size and speed up inference. This allows running surprisingly large models on modest hardware:

• Apple Silicon (M1/M2/M3/M4) — excellent performance via Metal. An M2 with 16GB runs Llama 3 8B at ~40 tokens/sec.
• NVIDIA GPU — CUDA support. An RTX 3060 (12GB) runs models up to 13B comfortably.
• CPU only — works, but slower (~5-10 tokens/sec on a modern processor for 7B).

The rule of thumb: 1GB of RAM per billion parameters (at 4-bit quantization). An 8B model needs ~8GB.

Ollama vs. alternatives

• vs. LM Studio — LM Studio has a native graphical interface and is friendlier; Ollama is lighter, CLI-first, and better for programmatic integration.
• vs. vLLM — vLLM is optimized for production/servers (high throughput); Ollama is for individual use/development.
• vs. Hugging Face — HF is a model repository; Ollama is the engine that runs them locally with simplicity.
• vs. cloud API (OpenAI, Anthropic) — the cloud has larger models (GPT-4, Claude) and requires no hardware; Ollama wins on privacy, zero cost, and offline.

Conclusion

Ollama has democratized access to state-of-the-art language models. In a world where data is the most valuable (and most vulnerable) asset, having the ability to run quality AI without depending on third parties is transformative — both for individual professionals and for organizations that take privacy seriously. If you haven't tried it yet, install it today: one command line is all that separates curiosity from autonomy.

References

• [1] Ollama — The easiest way to run open models — ollama.com
• [2] Ollama Docs — Quickstart & Installation — github.com/ollama/ollama
• [3] Open WebUI — ChatGPT-like interface for Ollama — github.com/open-webui
• [4] Ollama Model Library — ollama.com/library