While still investigating the various Docker storage options, I would like to save my Docker images/containers locally. To that end I need to set up a Docker Private Registry on my Nutanix virtualisation platform. Docker Hub provides a registry image ready for download that lets you have a private on-premise registry ready in a few commands. However, we want to be able scale our registry and use the availability features provided by the Acropolis management interfaces. For example, take advantage of things like redundant vDisks (RF2), snapshots and clones to take copies of the registry, and the ability to migrate the registry between hosts. In which case, the first thing to consider is how to persist the registry data volume on the Docker host VM. To this end I created a LVM stripe volume on top of mirrored vDisks as follows :
Add 6 x 100GB Nutanix VDisks to guest OS... [2:0:1:0] disk NUTANIX VDISK 0 /dev/sdb [2:0:2:0] disk NUTANIX VDISK 0 /dev/sdc [2:0:3:0] disk NUTANIX VDISK 0 /dev/sdd [2:0:4:0] disk NUTANIX VDISK 0 /dev/sde [2:0:5:0] disk NUTANIX VDISK 0 /dev/sdf [2:0:6:0] disk NUTANIX VDISK 0 /dev/sdg # pvcreate /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg Physical volume "/dev/sdb" successfully created Physical volume "/dev/sdc" successfully created Physical volume "/dev/sdd" successfully created Physical volume "/dev/sde" successfully created Physical volume "/dev/sdf" successfully created Physical volume "/dev/sdg" successfully created # vgcreate registry /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg Volume group "registry" successfully created # lvcreate -i 6 -l 100%VG -n registry registry Using default stripesize 64.00 KiB. Logical volume "registry" created. # mkfs.xfs /dev/mapper/registry-registry # mkdir -p /data/registry # grep registry /etc/fstab # Docker registry volume /dev/mapper/registry-registry /data/registry xfs defaults 0 0
The following command starts the registry with the newly created persistent storage:
docker run -d -p 5000:5000 --restart=always --name nx-registry \ -v /data/registry/:/var/lib/registry registry:2
This means I can now pull and push images to and from my private registry…
# docker pull couchbase Using default tag: latest latest: Pulling from library/couchbase b45376f323f5: Pull complete 23c388b926b6: Pull complete 10f1b5844a9c: Pull complete 2a7a952931ec: Pull complete fc38f156c0ea: Pull complete 815f08b3c781: Pull complete d9b2222c39b4: Pull complete 1a8da511d01b: Pull complete 5f37b8bdc5a6: Pull complete 18165b90fefa: Pull complete e8a83a5448df: Pull complete 5537747ea12f: Pull complete 30852bbad62b: Pull complete dd8c5611343d: Pull complete 45abdd57689a: Pull complete Digest: sha256:d6c6841c9ff7f00edc42c97a8023f013267924e040f1a1610d3638aa7d2fe9c2 Status: Downloaded newer image for couchbase:latest # docker tag couchbase localhost:5000/couchbase [root@fed22-docker-registry ~]# docker push localhost:5000/couchbase The push refers to a repository [localhost:5000/couchbase] (len: 1) 45abdd57689a: Image successfully pushed dd8c5611343d: Image already exists 30852bbad62b: Image already exists 5537747ea12f: Image already exists e8a83a5448df: Image successfully pushed 18165b90fefa: Image successfully pushed 5f37b8bdc5a6: Image successfully pushed 1a8da511d01b: Image successfully pushed d9b2222c39b4: Image already exists 815f08b3c781: Image successfully pushed fc38f156c0ea: Image already exists 2a7a952931ec: Image already exists 10f1b5844a9c: Image successfully pushed 23c388b926b6: Image successfully pushed b45376f323f5: Image successfully pushed latest: digest: sha256:36294aaf1bd29addb44bba2f28996aef8305435ee84b909c5149fd267548578c size: 32163 # docker pull localhost:5000/couchbase On the Docker VM host itself.... # pwd /data/registry/docker/registry/v2/repositories # ls couchbase nginx redis
This is fine if I only ever want to pull and push images from this local Docker host, but I may want to pull/push images to/from remote Docker clients. However, to do that securely would need a domain name that DNS would resolve to my registry host and a CA certificate. All of which costs me money, and for my lab system it’s a bit of a non-starter. So I am using self signed certificates and that means I need to configure each of my Docker clients with the created certificate.
Here’s a suggested self-signed certificate cli – make sure you use a FQDN and not an IP address for your CN entry, when creating the cert. For this to subsequently work you may, like me, need to create a dummy FQDN in the /etc/hosts file on each Docker VM host :
# mkdir -p /data/certs # openssl req -newkey rsa:4096 -nodes -sha256 -keyout /data/certs/domain.key -x509 \ -days 365 -out /data/certs/domain.crt cat /etc/hosts - needs to be added on every Docker VM 10.68.64.156 registry.nutanix.com
It’s then a case of copying the cert to my remote Docker clients. The default location on each Docker client host is /etc/docker/certs.d/registry.nutanix.com:5000 – on some Linux distro’s (Fedora?) you will need to create the directory structure below /etc/docker
/etc/docker/certs.d/registry.nutanix.com:5000/domain.crt
Make sure that you always restart the Docker daemon on each host so that any config or cert changes get picked up
systemctl stop docker systemctl daemon-reload systemctl start docker
Now that we have TSL configured, we can look at setting up some basic authentication. Steps are …
mkdir /data/auth
docker run --entrypoint htpasswd registry:2 -Bbn ray ********** > /data/auth/htpasswd
This should allow me as user ray to login to the registry and run docker pulls/pushes etc
docker-client# docker login registry.nutanix.com:5000 Username: ray Password: ********** Email: ray@nutanix.com WARNING: login credentials saved in /root/.docker/config.json Login Succeeded docker-client# docker pull registry.nutanix.com:5000/couchbase docker-client# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE 10.68.64.156:5000/couchbase latest c10532c5c520 2 weeks ago 372 MB
Now that the Docker command line needed to start our registry with persistent volumes, TLS, basic authentication etc, has begun to grow. It’s probably advisable to consider using Docker Compose. We can install it quite quickly as follows (see Docker docs for other install methods):
# curl -L https://github.com/docker/compose/releases/download/1.5.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
# chmod +x /usr/local/bin/docker-compose
My docker-compose.yml file now looks like:
# cat docker-compose.yml registry: restart: always image: registry:2 ports: - 5000:5000 environment: REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt REGISTRY_HTTP_TLS_KEY: /certs/domain.key REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm volumes: - /data/registry:/var/lib/registry - /data/certs:/certs - /data/auth:/auth
and we can start the registry by running …
docker-compose up -d
So that’s it, a private on-premise registry backed by local storage in the guise of the Acropolis Distributed Storage Fabric (DSF). No need for any local Redis caching that you might consider for cloud backed storage etc. Next steps for me are to take a snapshot of my new registry VM and maybe even migrate it to another host. Other things to consider might be improved authentication (using nginx?) perhaps. In future posts I intend to outline how an ecosystem of cloud aware apps like CouchDB etc are spun up in matter of minutes, I am also keen to look at how to use Docker persistent volumes in the monitoring of logs, etc.
Additional Info
http://docs.docker.com/registry/deploying/
https://docs.docker.com/compose/install/