Hello Neural Explorers!
It’s time to set up your Docker development environment.
Go into your home directory.
cd $HOME
Check if you have an SSH key.
ls .ssh
If you have an id_rsa.pub file, it’s okay. If not, create one using ssh-keygen.
Create a directory to store models on your host. Create also a directory to configure your Docker Compose environment. Do not modify the directory name afterwards since Compose uses it to name its containers.
mkdir models
mkdir llmdev
cd llmdev
Copy your ssh public key into your Docker Compose environment.
mkdir .ssh
cp $HOME/.ssh/id_rsa.pub .ssh/authorized_keys
Create a .env file.
MODELS=$HOME/models
Create a Dockerfile file. If you want to use specific Linux packages within your container, you can include them in that file.
FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 as build
RUN apt-get update && apt-get install -y \
coreutils \
procps \
iproute2 \
iputils-ping \
openssh-server \
curl \
git \
vim \
python3 \
python3-pip
RUN groupadd -g 1000 dev && \
useradd -rm -d /home/dev -s /bin/bash -g dev -u 1000 dev && \
chown -R dev:dev /home/dev
USER 1000
COPY --chown=1000 . /home/dev
WORKDIR /home/dev
RUN pip install --no-warn-script-location --upgrade pip setuptools wheel \
&& pip install --no-warn-script-location -r requirements.txt
USER root
RUN mkdir -p /run/sshd
CMD ["/usr/sbin/sshd","-D"]
Create a docker-compose.yml file. This configuration file binds the container’s SSH port to 2222 on your local machine. If that port is already being used by another application, feel free to adjust accordingly. Additionally, you may choose to bind additional ports as needed. It also uses the host’s SSH configuration so that everytime you build your container, it will always have the same host signature.
version: "3.8"
services:
llmdev:
build:
context: .
env_file:
- .env
volumes:
- devdata:/home/dev
- type: bind
source: /etc/ssh
target: /etc/ssh
- type: bind
source: $MODELS
target: /models
ports:
- "2222:22"
networks:
- dev-network
container_name: llmdev
hostname: llmdev
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
devdata:
networks:
dev-network:
driver: bridge
Create a requirements.txt file. If you want to use specific Python packages within your container, you can include them in that file.
accelerate --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ auto-gptq bitsandbytes datasets optimum protobuf scipy sentencepiece torch transformers
Edit (March 2024) : Not setting the versions in the requirements file is a bad practice that should not be reproduced in production. The issue is that the libs evolve very quickly and if you set the versions, you will not necessarily be able to load the latest models. When I wrote this lesson at the end of 2023 everything was working fine. A few weeks later, there was an incompatibility in the libs and we had to choose the correct versions. And again a few weeks later, everything was working again. I will add a lesson for using a docker image with TGI (Text Generation Inference) from Hugging Face. This will make it easier to launch a model without worrying about libs.
Now you’re ready to start building your container.
docker-compose build
Wait 5 minutes or longer depending on your connection and run it.
docker-compose up -d
Connect to your container. As your SSH private key has been added to the container’s authorized_keys file, no password prompt should be displayed.
ssh dev@localhost -p2222
You can code directly inside your container using VIM. However, It would be much more convenient to use Visual Studio Code. To do so, install the “Visual Studio Code Remote – SSH” extension. Now, you have everything you need to get started coding. In the next lesson, you will learn how to create your first LLM app.
