Install your dev container

It’s time to set up your Docker development environment.

Go into your home directory.

cd $HOME

Check if you have an SSH key.

ls .ssh

If you have an id_rsa.pub file, it’s okay. If not, create one using ssh-keygen.

Create a directory to store models on your host. Create also a directory to configure your Docker Compose environment. Do not modify the directory name afterwards since Compose uses it to name its containers.

mkdir models
mkdir llmdev
cd llmdev

Copy your ssh public key into your Docker Compose environment.

mkdir .ssh
cp $HOME/.ssh/id_rsa.pub .ssh/authorized_keys

Create a .env file.

MODELS=$HOME/models

Create a Dockerfile file. If you want to use specific Linux packages within your container, you can include them in that file.

FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 as build

RUN apt-get update && apt-get install -y   coreutils   procps   iproute2   iputils-ping   openssh-server   curl   git   vim   python3   python3-pip

RUN groupadd -g 1000 dev &&   useradd -rm -d /home/dev -s /bin/bash -g dev -u 1000 dev &&   chown -R dev:dev /home/dev

USER 1000

COPY --chown=1000 . /home/dev

WORKDIR /home/dev

RUN pip install --no-warn-script-location --upgrade pip setuptools wheel     && pip install --no-warn-script-location -r requirements.txt

USER root

RUN mkdir -p /run/sshd

CMD ["/usr/sbin/sshd","-D"]

Create a docker-compose.yml file. This configuration file binds the container’s SSH port to 2222 on your local machine. If that port is already being used by another application, feel free to adjust accordingly. Additionally, you may choose to bind additional ports as needed. It also uses the host’s SSH configuration so that everytime you build your container, it will always have the same host signature.

version: "3.8"
services:
  llmdev:
    build:
      context: .
    env_file:
      - .env
    volumes:
      - devdata:/home/dev
      - type: bind
        source: /etc/ssh
        target: /etc/ssh
      - type: bind
        source: $MODELS
        target: /models
    ports:
      - "2222:22"
    networks:
      - dev-network
    container_name: llmdev
    hostname: llmdev
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  devdata:
networks:
  dev-network:
    driver: bridge

Create a requirements.txt file. If you want to use specific Python packages within your container, you can include them in that file.

accelerate
--extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
auto-gptq
bitsandbytes
datasets
optimum
protobuf
scipy
sentencepiece
torch
transformers

Edit (March 2024) : Not setting the versions in the requirements file is a bad practice that should not be reproduced in production. The issue is that the libs evolve very quickly and if you set the versions, you will not necessarily be able to load the latest models. When I wrote this lesson at the end of 2023 everything was working fine. A few weeks later, there was an incompatibility in the libs and we had to choose the correct versions. And again a few weeks later, everything was working again. I will add a lesson for using a docker image with TGI (Text Generation Inference) from Hugging Face. This will make it easier to launch a model without worrying about libs.

Now you’re ready to start building your container.

docker-compose build

Wait 5 minutes or longer depending on your connection and run it.

docker-compose up -d

Connect to your container. As your SSH private key has been added to the container’s authorized_keys file, no password prompt should be displayed.

ssh dev@localhost -p2222

You can code directly inside your container using VIM. However, It would be much more convenient to use Visual Studio Code. To do so, install the “Visual Studio Code Remote – SSH” extension. Now, you have everything you need to get started coding. In the next lesson, you will learn how to create your first LLM app.