Ollama

Ollama is the easiest way to run open-source large language models locally, packaging model download, quantization, and serving into a single CLI tool — providing a Docker-like experience where `ollama pull llama3` downloads a model and `ollama run llama3` starts an interactive chat session, with a built-in OpenAI-compatible REST API that enables local LLM integration into any application without cloud API costs, internet dependency, or data privacy concerns.

What Is Ollama?

• Definition: A local LLM runtime that wraps llama.cpp in a user-friendly package — handling model downloading, GGUF format management, GPU detection, memory allocation, and API serving so users never interact with raw model files or compilation flags.
• One-Line Install: `curl -fsSL https://ollama.com/install.sh | sh` on Linux/Mac — a single command installs the Ollama daemon, CLI, and all dependencies. Windows installer also available.
• Docker-Like Model Management: `ollama pull` downloads models, `ollama list` shows installed models, `ollama rm` removes them — the same mental model as Docker images, making it immediately familiar to developers.
• Model Library: Ollama hosts a curated library of pre-quantized models — Llama 3, Mistral, Mixtral, Phi-3, Gemma, CodeLlama, Qwen, Command R, and dozens more, each available in multiple size variants (7B, 13B, 70B) and quantization levels.
• OpenAI-Compatible API: `http://localhost:11434/v1/chat/completions` — applications using the OpenAI SDK can switch to local inference by changing the base URL, with zero code changes to the application logic.

Key Features

• Automatic GPU Detection: Ollama detects NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal) GPUs automatically — no manual CUDA configuration or driver management.
• Model Customization (Modelfile): Create custom model configurations with system prompts, temperature settings, and parameter overrides — `FROM llama3` + `SYSTEM "You are a helpful coding assistant"` creates a specialized variant.
• Concurrent Requests: The Ollama server handles multiple simultaneous requests with automatic batching — suitable for multi-user development teams sharing a single GPU server.
• Embedding API: `ollama.embeddings(model="nomic-embed-text", prompt="text")` generates embeddings locally — enabling fully local RAG pipelines without any cloud API calls.
• Multimodal: Support for vision models (LLaVA, Llama 3.2 Vision) — send images alongside text prompts for local multimodal inference.

Ollama Model Library (Popular Models)

Ollama vs Alternatives

Ollama is the tool that made local LLM inference as simple as running a Docker container — wrapping the complexity of model management, quantization, and GPU configuration into a familiar pull/run workflow with an OpenAI-compatible API that lets developers build privacy-preserving AI applications without cloud dependencies.