Docker for AI apps

Problem Docker solves

You build a project, it works on your machine, you deploy it to a server, and it crashes.

The reason is almost always the environment. Your project depends on libraries like NumPy, pandas, PyTorch, LangChain and FastAPI, and each of those depends on other libraries, and each one is built by a different team that releases new versions on its own schedule. A new version of one library can break another one. On your machine you fixed that months ago by upgrading something, and you forgot. On the server, a fresh install picks up different versions, a different Python, or a different operating system, and the same code fails.

You fix one error, deploy again, and get another one: this package is not compatible with that package. This can go on for hours.

Docker solves it by shipping the environment together with the code. Instead of sending your code and hoping the server can rebuild the same environment, you build the environment once, as an image, and every machine runs exactly that image.

Images and containers

Two words come up constantly, and they mean different things.

An image is the package: a read-only template that contains an operating system base, Python, your libraries, your code and the command to start it. You build it once from a Dockerfile.

A container is a running instance of an image. You can start many containers from one image, on any machine that has Docker, and each runs isolated from the others and from the host.

If you know Python classes, an image is like a class and a container is like an object created from it.

A container is similar to a virtual environment, in the sense that it holds everything a project needs, but it goes further. A virtual environment only isolates Python packages, while a container isolates the whole system: the operating system libraries, the Python version, the file system and the network.

Installing Docker

On macOS and Windows, install Docker Desktop, which runs the Docker engine in the background and gives you a dashboard of your images and containers. On Linux, install Docker Engine. Docker Desktop and the images you build take a lot of disk space, so keep an eye on it with docker system df and clean up old images with docker system prune.

A Dockerfile for an AI API

A Dockerfile is a script that tells Docker, step by step, how to build the image for your project.

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

RUN useradd --create-home appuser
USER appuser

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Line by line:

• FROM python:3.12-slim starts from an official image with Python already installed. The slim variant leaves out tools you don't need, which keeps the image smaller and faster to build and deploy.
• The two ENV lines stop Python from writing .pyc files and make your logs appear immediately instead of being buffered.
• WORKDIR /app sets the folder inside the container where everything happens.
• COPY requirements.txt . and the pip install line install your dependencies. They come before copying the code on purpose, which the next section explains.
• COPY app/ ./app/ copies your code.
• useradd and USER run the app as a normal user instead of root, so if someone breaks into the container, they have fewer permissions.
• EXPOSE 8000 documents the port the app listens on.
• CMD is the command that starts the app when the container runs. --host 0.0.0.0 is required, because the default 127.0.0.1 only accepts connections from inside the container itself.

Build the image and run a container from it:

docker build -t rag-api .
docker run --rm -p 8000:8000 --env-file .env rag-api

-t rag-api names the image, -p 8000:8000 connects port 8000 on your machine to port 8000 in the container, --env-file .env passes your environment variables at runtime, and --rm deletes the container when it stops.

Coding agents like Claude Code can write a Dockerfile for you in seconds, and that is fine, but read every line and understand why it is there, because when the build fails on the server, you are the one who has to fix it.

Layer caching

Each instruction in a Dockerfile creates a layer, and Docker caches layers. When you rebuild, Docker reuses every layer up to the first one that changed, and rebuilds everything after it.

That is why requirements.txt is copied and installed before the code. Your code changes on every build, but your dependencies change rarely, so with this order, Docker reuses the slow pip install layer and only re-copies your code, and a rebuild takes seconds instead of minutes. If you copy all the code first and then install, every small code change reinstalls every library.

Pin your dependency versions in requirements.txt, so a rebuild next month installs the same versions as today.

.dockerignore file

Like .gitignore, a .dockerignore file keeps files out of the image.

.env
venv/
.venv/
__pycache__/
.git/
data/
models/
logs/
*.ipynb

Without it, COPY can send your virtual environment, your Git history, your datasets and, most importantly, your .env file with real keys into the image, and anyone who gets the image gets the keys.

Secrets and configuration

Never put API keys in the Dockerfile, and never copy the .env file into the image. An image is a file that gets pushed to registries, shared and cached, and anything inside it can be read.

Pass secrets when the container starts, with --env-file or -e on your machine, and with the secret settings of your platform in production. The image stays the same in every environment, and only the configuration changes.

Volumes for models and data

Models and datasets are large, and baking them into the image makes it huge and slow to push and pull. A volume mounts a folder from the host machine, or a Docker-managed storage area, into the container.

docker run --rm -p 8000:8000 \
  --env-file .env \
  -v $(pwd)/models:/app/models \
  -v hf-cache:/home/appuser/.cache/huggingface \
  rag-api

The first -v mounts your local models folder into the container, and the second creates a named volume for the Hugging Face cache, so a model downloaded once is reused the next time the container starts instead of being downloaded again.

Volumes are also how data survives a container restart. Anything a container writes to its own file system disappears when the container is removed.

Docker Compose for an API plus a vector database

Most AI applications are more than one service: an API, a vector database, maybe a cache like Redis, a worker that processes jobs, and a frontend. Each one has its own environment and dependencies, and they need to talk to each other.

Docker Compose describes all of them in one YAML file and starts them together with one command, so the whole system runs the same way for every developer, whether they use macOS, Windows or Ubuntu.

services:
  api:
    build: .
    ports:
      - "8000:8000"
    env_file: .env
    environment:
      QDRANT_URL: http://qdrant:6333
    depends_on:
      - qdrant

  qdrant:
    image: qdrant/qdrant
    ports:
      - "6333:6333"
    volumes:
      - qdrant-data:/qdrant/storage

volumes:
  qdrant-data:

docker compose up --build
docker compose logs -f api
docker compose down

Inside Compose, services reach each other by their service name, so the API connects to http://qdrant:6333, not localhost. That is one of the most common mistakes: inside a container, localhost means the container itself, not your machine and not the other services.

Sandboxing code that agents run

Docker is also a security tool for AI. When an agent writes code and runs it, or a user uploads a file to process, you don't want that code running on your server with access to your files, your network and your keys.

Running it inside a container, with no access to secrets, limited CPU and memory, no network or a restricted one, and a time limit, is called sandboxing. If the code does something harmful, the damage stays inside a container that you throw away afterwards.

Docker also has a Python SDK, so your application can start, run and remove these sandbox containers from code.

import docker

client = docker.from_env()
output = client.containers.run(
    "python:3.12-slim",
    ["python", "-c", "print(sum(range(10)))"],
    network_disabled=True,
    mem_limit="256m",
    remove=True,
)
print(output.decode())

GPUs in containers

If your container runs a local model on an NVIDIA GPU, the host needs the NVIDIA drivers and the NVIDIA Container Toolkit, and you start the container with --gpus all. Use a base image that matches your CUDA version, like the official PyTorch or NVIDIA CUDA images, rather than installing CUDA yourself. GPU images are large, often several gigabytes, so this is where volumes for model weights and careful layering matter most.

Keeping images small

• Start from a slim base image.
• Use --no-cache-dir with pip.
• Keep models and data out of the image, and mount them as volumes.
• Use a .dockerignore file.
• Install only what the app needs at runtime, and leave test and notebook libraries out of the production requirements.
• For complex builds, use a multi-stage Dockerfile that builds in one stage and copies only the result into a clean final stage.

Commands you will use most

docker build -t name .
docker run --rm -p 8000:8000 --env-file .env name
docker ps
docker ps -a
docker logs -f <container>
docker exec -it <container> bash
docker stop <container>
docker images
docker system df
docker system prune

docker exec -it <container> bash opens a terminal inside a running container, which is the fastest way to check whether a file, a package or an environment variable is really there.

From the floor

In the 75 day challenge, Docker was day 7, the last of the foundation days, and day 8 was Docker Compose. I explained Compose with a simple example from a subscription product: when a free user pays and becomes a paid member, that information passes through several services, each with its own environment and libraries, and Compose is what makes all of them run together the same way on everyone's machine.

In the first hands-on session of AI Engineer HQ cohort 1, we started with sandboxing and containers before any AI code. We wrote the Dockerfile line by line, built an image called ai-sandbox, ran a script inside it, and then started containers from Python with the Docker SDK. I also showed Hugging Face Spaces, which uses Docker to host models and demos, as an example of containers in the AI world you already use.

Related

• Cloud basics: your first deploy
• Environment variables and secrets
• Linux and the terminal
• LLMOps

Sources

• Docker overview, Docker documentation
• Dockerfile best practices, Docker documentation
• Docker Compose documentation
• Docker Spaces, Hugging Face
• NVIDIA Container Toolkit