Inspecting a Private Docker Container Registry

When I started to work with containers, most of the images I wanted to use were available on DockerHub, some others also on quay.io. If I want to browse through available images, version tags, or find out when a specific image was released, I can usethese service’s websites. However, it won’t take long until you will be tasked to work with images in private registries, in example a registry that your development team uses in the build pipeline. In such a situation, an easy-to-use web app to manage images is not always available. In this post I want to document the different ways to grab image meta data from a registry through the standardized OCI Docker Registry HTTP API V2.

Deploy a Local Private Docker Registry

To test out the different ways of communicating with a private registry, let’s setup or own local container image registry. All you need is a working installation of Docker. You should also be able to replace all docker commands with podman (although I haven’t tested it).

The Docker Registry is kind of touchy when it comes to using plain HTTP listeners. Therefore we’ll setup a secure registry with a self-signed certificate.

The quickest method to setup a container registry is running it as a Docker container, which is also described in the docs. We want to configure the container with some custom parameters to make it resemble a production-grade registry deployment as good as possible:

• Provide the self-signed certificate
• Configure user and password authentication
• Make pushed images persistent to survive container restarts

Generate Certificates

For generating the self-signed certificate we will follow the guidance of the Docker documentation. At first we will create a separate directory for the certificates. This will simplify mounting the certificates in the Docker Registry container later on. Also, we need to add CN=localhost to the subject field to make the hostname match with the hostname we have to use later on to communicate with registry:

# Create a separate folders for the certificates 
mkdir -p certs

# Generate a self-signed certificate
openssl req \
    -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
    -x509 -days 365 -out certs/domain.crt \
    -subj "/CN=localhost/C=DE/ST=BW/L=Stuttgart/O=Foo/OU=Bar"

Generate HTTP Basic Authentication File

The command htpasswd required in order to generate a file for configuring basic authentication is not installed by default on some Linux machines. If htpasswd is not installed on your box, you can choose option B using a very small and well-maintained Apache2 container image, which has htpasswd on board:

# Create a separate folder for the auth file
mkdir -p httpauth

# Option A: Generate the auth file with native htpasswd command (if installed)
htpasswd -Bbn myuser mypass123 > httpauth/htpasswd

# Option B: Generate the auth file with htpasswd from the Docker Registry container
docker run \
  --rm \
  --entrypoint htpasswd \
  httpd:2-alpine -Bbn myuser mypass123 > httpauth/htpasswd && echo "Auth file generated."

The Docker docs explicitly state how to use htpasswd. Don’t be sloppy with the required arguments (-Bbn).

Deploy the Registry Container

Now we can deploy the registry with our custom configuration. Additionally to the command line suggested by the docs we will add a volume mount in order to persist any images pushed to the registry into a directory on the host. For this purpose we will create a separate directory and later on mount it .

mkdir -p registry

Now we can launch the registry container:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v "$(pwd)"/registry:/var/lib/registry \
  -v "$(pwd)"/httpauth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v "$(pwd)"/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

Before continueing we should check whether the container is running properly and make sure that no error messages are logged for the registry container:

sudo docker ps -a

… should show a running registry:

CONTAINER ID   IMAGE        [...]   STATUS              [...]   NAMES
04da5fb466b3   registry:2   [...]   Up About a minute   [...]   registry

Login to the Registry

Use docker login to store the basic authentication credentials in your home folder:

docker login localhost:5000 -u myuser -p mypass123

Push Images Into the Registry

To test whether our registry works and is accessible, we will tag one of the images we have already pulled onto the local machine and push it to our local registry:

# Tag the httpd image
docker tag httpd:2-alpine localhost:5000/httpd:2-alpine
docker tag httpd:2-alpine localhost:5000/httpd:latest

# Push it to the local registry
docker push localhost:5000/httpd:2-alpine
docker push localhost:5000/httpd:latest

The following code snippet would automatically push all locally available images to the registry:

# Get all image names except the already tagged ones
images=$(docker images | sed 1d | awk -F ' ' '{print $1 ":" $2}' | grep -v 'localhost:5000')

# Tag and push all images in the list
for name in $images; do
  docker tag $name localhost:5000/$name
  docker push localhost:5000/$name
done

API Requests With cURL

The first method to cover here will be standard HTTP requests. Since I am on a Linux system, we can use the cURL tool to send HTTP requests to the registry’s API.

Authentication

For the reason that the registry is secured with a basic authentication and the self-signed certificate, we need to provide this information on executing cURL. Here is an example:

curl \
  --user myuser:mypass123 \
  --cacert certs/domain.crt \
  --write-out "%{http_code}\n" \
  https://localhost:5000

If this returns 200, this indicates that HTTP authentication and TLS-encrypted communication works as expected – nice!

We can reuse most of this request for other requests, so let’s put the fixed parts into a shell function. To define this function, paste and execute all line at once:

cr_api() {
  curl \
  --silent \
  --user myuser:mypass123 \
  --cacert certs/domain.crt \
  https://localhost:5000/$1 | jq -r $2
}

As you may have noticed, the function also uses the jq command. It is used to cleanly format the returned JSON payload. Find installation instructions here. If you cannot or don’t want to install jq, simply remove the jq command from the function:

cr_api() {
  curl \
  --silent \
  --user myuser:mypass123 \
  --cacert certs/domain.crt \
  https://localhost:5000/$1
}

Sample Requests

Get All Repositories

With our previously defined function cr_api() we can request a list of image repositories in the registry:

cr_api v2/_catalog

Returns the JSON output:

{
  "repositories": [
    "httpd",
    "registry"
  ]
}

With the capability of using jq for processing the JSON output we can get all repositories as a list:

# Request:
cr_api v2/_catalog .repositories[]

# Result:
httpd
registry

Get All Tags for a Repository

# Request:
cr_api v2/httpd/tags/list

# Result:
2-alpine
latest

Advanced Requests

The APi let’s you basically do all the same things that you can accomplish with the docker commands. Check out the Docker docs for more details. Stuff that you can implement in example if you like to:

• Deleting layers from the registry
• Pushing image manifests
• Getting manifests in order to determine whether an image has already been pulled

Registry Exploration With Skopeo

A more comfortable method for communicating the API is using Skopeo:

Skopeo is a command line utility that performs various operations on container images and image repositories. […] Skopeo works with API V2 container image registries such as docker.io and quay.io registries, private registries, local directories and local OCI-layout directories. […]

Skopeo GitHub Project

Installation

skopeo is available for all major Linux distribution. Find further installation instruction here. As I’m using Ubuntu, this is how to install the latest version of skopeo on Ubuntu accoring to the official docs (including versions >20.10):

. /etc/os-release
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install skopeo

Authentication

Like for the classic docker command, for skopeo we need to login to the registry before we can use the API. In our specific case here, we need to tweak stuff a little bit beforehand:

• skopeo doesn’t know about the local certificate we generated earlier. We need to point skopeo to the certificate directory for each execution.
• skopeo by default looks for a certificate with a file extension *.cert instead of *.crt. We’ll create soft link to fix this.
• skopeo is written in Go. In the current versions of Go, the type of certificates we generated earlier are deprecated as we didn’t include Subject Alternative Names (SANs). The common workaround is to set the GODEBUG environment variable.

if you are working with a production registry which uses signed certificates, all of these three preparations steps become obsolete. But for our local test scenario this is a good opportunity to document these steps:

# Tell skopeo to accept certificates without SANs
export GODEBUG=x509ignoreCN=0

# Create a soft link to allow skopeo to find the certificate using file name 'domain.cert'
ln -s $(realpath certs)/domain.crt $(realpath certs)/domain.cert

# Attempt to login to the registry with skopeo with user and password and also the certificate
skopeo login localhost:5000 -u myuser -p mypass123 --cert-dir=./certs/

Now, you should see: Login Succeeded!

If we would skip above step setting GODEBUG=x509ignoreCN=0, we would have gotten the following error message:

FATA[0000] [...] Get "https://localhost:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

For local testing purposes it is not necessary to verify the self-signed TLS certificate delivered by the registry. To disable TLS verification implicitly, we can create an alias for the skopeo command:

alias skopeo="skopeo --tls-verify=false"

Please note that setting the GODEBUG and the shell alias are only persistent to your current shell. If you want to persist these two settings, you need to add them to your user’s profile file:

echo 'alias skopeo="skopeo --tls-verify=false"' > ~/.profile
echo 'GODEBUG=x509ignoreCN=0' > ~/.profile

Usage Examples

To be specific and correct with the used terms: The image httpd we pushed earlier, creates a repository with name httpd in the registry. For this repository, we created two tags: 2-alpine and latest.

Get All Repositories

It looks like that skopeo does not support this by the time I am writing this post. From making a request with skopeo inspect docker://localhost:5000 I expected to get a list of repositories available in the registry. Instead it fails with an authentication required error message. But this is not a big deal – we can simply use the cURL request we covered earlier in this post in order to obtain the available repositories.

Get Repository Details

Let’s get the details of a specific repository we previously pushed to the registry:

skopeo inspect docker://localhost:5000/httpd

This gives us the details of the repository httpd tagged with 2-alpine:

{
    "Name": "localhost:5000/httpd",
    "Digest": "sha256:28e81dcc1e65afafc88067ef9839233c56af72faf2aa4cba529fbd5d7fb81987",
    "RepoTags": [
        "2-alpine",
        "latest"
    ],
    "Created": "2020-12-17T13:26:10.442999769Z",
    "DockerVersion": "19.03.12",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:801bfaa63ef2094d770c809815b9e2b9c1194728e5e754ef7bc764030e140cea",
        "sha256:ac8f86b44b1772588be35f990817bd331f91e0bf0faf199c3f3725965871183e",
        "sha256:078b6c86de97046a30960a9746e9bff234739107f624b91c61beb637982794f1",
        "sha256:55f318a9c48a0edd5526841012fa7dc71a0a431b8648bd2f5235b9436e1ddc4f",
        "sha256:5da5afdb6ea0250774cbbb925bfd56194d355172618e3ba033dc15993e8ece25"
    ],
    "Env": [
        "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "HTTPD_PREFIX=/usr/local/apache2",
        "HTTPD_VERSION=2.4.46",
        "HTTPD_SHA256=740eddf6e1c641992b22359cabc66e6325868c3c5e2e3f98faf349b61ecf41ea",
        "HTTPD_PATCHES="
    ]
}

Important: I figured out that skopeo can only return details on an image, in case a tag with name latest is present in the registry if you don’t add a specific tag to the request. Otherwise the query will fail with message:

FATA[0000] Error parsing image name "docker://localhost:5000/httpd": Error reading manifest latest in localhost:5000/httpd: manifest unknown: manifest unknown

Get Even More Details of an Image

While we were able to list some generic meta data for the image with the previous command, we can also get detailed information like:

• commands used to build the different layers of an image in the original Dockerfile.
• ports exposed by the container image by default.
• environment variables defined for the image.

It is sufficient to add the --config argument to obtain these details for this purpose. In addition it makes sense to pipe the resulting output through jq as the JSON output is not pretty-formatted by default:

skopeo inspect --config docker://localhost:5000/httpd:2-alpine | jq

This gives us an output like this (shortened):

{
  "created": "2020-12-17T13:26:10.442999769Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "ExposedPorts": {
      "80/tcp": {}
    },
    "Env": [
      "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "HTTPD_PREFIX=/usr/local/apache2",
      "HTTPD_VERSION=2.4.46",
      "HTTPD_SHA256=740eddf6e1c641992b22359cabc66e6325868c3c5e2e3f98faf349b61ecf41ea",
      "HTTPD_PATCHES="
    ],
    "Cmd": [
      "httpd-foreground"
    ],
    "WorkingDir": "/usr/local/apache2",
    "StopSignal": "SIGWINCH"
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:777b2c648970480f50f5b4d0af8f9a8ea798eea43dbcf40ce4a8c7118736bdcf",
      "sha256:d976dde9f1cce7df708d5526eb92b789b41742ab0df4857395135edcfc36d83a",
      "sha256:4be096a5acf0fb23f012fac5a27d0d8d6192fdadc889eaabd4e5cc3904d2ac31",
      "sha256:36bbebca5737e697db2256ab78c7a99bc073e8fcc7860be094828390e9f78176",
      "sha256:386a4a1cb9008243e5fbeca8acec2e836df8d7e377479df846a3b20949aa4d1b"
    ]
  },
  "history": [
    {
      "created": "2020-12-17T00:19:41.960367136Z",
      "created_by": "/bin/sh -c #(nop) ADD file:ec475c2abb2d46435286b5ae5efacf5b50b1a9e3b6293b69db3c0172b5b9658b in / "
    },
    {
      "created": "2020-12-17T00:19:42.11518025Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
      "empty_layer": true
    },
    {
      "created": "2020-12-17T13:24:48.412976922Z",
      "created_by": "/bin/sh -c set -x \t&& addgroup -g 82 -S www-data \t&& adduser -u 82 -D -S -G www-data www-data"
    },
    {
      "created": "2020-12-17T13:24:48.590085431Z",
      "created_by": "/bin/sh -c #(nop)  ENV HTTPD_PREFIX=/usr/local/apache2",
      "empty_layer": true
    },
 [...]
    {
      "created": "2020-12-17T13:26:10.442999769Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"httpd-foreground\"]",
      "empty_layer": true
    }
  ]
}

The original output contains a few more layers, which I removed for the above excerpt.

Compatibility With Public Registries

You can use the same commands for inspecting repositories in commonly known public registries as well.

skopeo list-tags docker://docker.io/nginx

… lists all available tags for the nginx repository (output shortened):

{
    "Repository": "docker.io/library/nginx",
    "Tags": [
        "1-alpine-perl",
        "1-alpine",
        "1-perl",
        "1.10-alpine",
        "1.10.0-alpine",
        "1.10.0",
        "1.10.1-alpine",
        "1.10.1",
        "1.10.2-alpine",
        "1.10.2",
        "1.10.3-alpine",
 [...]
        "alpine-perl",
        "alpine",
        "latest",
        "mainline-alpine-perl",
        "mainline-alpine",
        "mainline-perl",
        "mainline",
        "perl",
        "stable-alpine-perl",
        "stable-alpine",
        "stable-perl",
        "stable"
    ]
}

Querying Local File Systems

After a container runtime like Docker or Podman has pulled an image from a registry, its stored on the local file system of the runtime. This is not a registry, but the local storage of the runtime from which it start the images as containers. Skopeo a can also inspect these images.

Let’s pull the latest busybox image using the Docker daemon:

docker pull busybox:latest

We can now inspect this image using:

skopeo inspect --config docker-daemon:busybox:latest | jq

… which yields:

{
  "created": "2020-12-30T01:19:53.606880456Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "sh"
    ]
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:1dad141bdb55cb5378a5cc3f4e81c10af14b74db9e861e505a3e4f81d99651bf"
    ]
  },
  "history": [
    {
      "created": "2020-12-30T01:19:53.451174889Z",
      "created_by": "/bin/sh -c #(nop) ADD file:15a162eb8a80b48c798f86562d2cc59c017f6b887907a5332996a3723f5aa1fa in / "
    },
    {
      "created": "2020-12-30T01:19:53.606880456Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"sh\"]",
      "empty_layer": true
    }
  ]
}

A notice regarding permissions: As of today, Docker installations by default run as root. If the user you are operating with on the shell is not member of the root group, you have to add the sudo command to allow skopeo to connect to the local Docker daemon:

skopeo inspect --config docker-daemon:busybox:latest | jq

Otherwise you would get an error message similar to this one:

FATA[0000] [...] Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock [...]

Conclusion

To me it seems hard for beginners to work with private container image registries, because there is a diverse tool set out there, whereat you cannot cover all the usual tasks with one of these tools. Public services like DockerHub or Quay.io are often times the first contact point for beginner, because they offer an easy-to-use user experience to browse through and manage repositories using their web apps.

For deploying workloads on major clouds, there are also managed private registry services like Azure Container Registry (Azure) or Elastic Container Registry (AWS), which provide rich UIs, CLI tools and APIs as well. If you rely on these kind of highly integrated services, the guidance in this blog post may not be of much help for you.

If you’re working with containers in a an enterprise environment or if you want to build a local development environment on your system resembling the real world as close as possible, this blog post should provide some helpful guidance setting up and communicating with a private registry.

To dive deeper into skopeo I can recommend the release announcement blog post for version 1.0 in the RedHat blog.

This wraps it up for today. Hope the readers can take away little bit of new knowledge helping them with their daily work.