Installing Docker CE Despite Broken Ubuntu Package Lists

Yes – there are still people out there using Docker 🙂 A few days ago I experienced some trouble getting Docker installed on a AWS-hosted EC2 instance running on Ubuntu 18.04. On normal days the install scripts for this kind of box automatically installs a Docker version compatible with the Kubernetes Engine which I use on that system (which is by the way the great RKE with Rancher). But that day was not a normal day …

And this is where it all started …

It seemed that the box got stuck at some point during the install process in my script. In the course of setting up the box it tries to launch a Docker container, but complains that docker is not installed. To troubleshoot, I tried to track down the error cause by running the installation manually using the Ubuntu repositories. But even before I was able to install any package, apt update smacked my face with a strange error when I tried to update the Ubuntu package lists.

marc@foo:~# sudo apt update
Hit:1 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
[...]
Err:7 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
  File has unexpected size (9051 != 7889). Mirror sync in progress? [IP: 143.204.101.104 443]
[...]
E: Some index files failed to download. They have been ignored, or old ones used instead.

I had my explanation for the missing Docker installation: apt was unable to acquire the package lists and hence could not run the installation of the Docker package.

I’ve never had a problem like this before, where the remote repositories were not working, because the downloaded file including the list was somehow broken.

The first lesson: External services and repositories might not be available all the time. If your basic install process for your environment requires these services to come alive, always try to provide a plan B to workaround situations in which the external service is not responding or not working correctly. And if plan B isn’t working and your environment is too crucial to fail, have something like a plan C that gets you out of the country with a new identity.

But how to solve this particular problem with the broken package list?

How to manually download and install the required packages

As I’ve been working with Ubuntu now for more that 11 years, I had to figure out more than once a different way to install specific packages than by using apt. You can also install built-and-ready applications without using apt install. By downloading the DEB packages manually and running dpkg on them, you basically perform the same procedure like apt install. The little difference here is, that in this case you need to know which packages your main application package is dependent on, as apt normally does that for you.

Fortunately, the guys from Docker describe these dependencies in their online manual. In my example I want to install Docker CE 19.03.3. After downloading the DEB package I tried to install it:

dpkg -i docker-ce_19.03.3~3-0~ubuntu-bionic_amd64.deb

Running this command, dpkg for this exact package will complain about missing dependencies, as we need to install these dependencies before we install the main package. It also tells you which exact version the dependencies need to have at a minimum. Accordingly, I downloaded the two additional packages which dpkg was asking for. So at the bottom line, you need three packages in our specific case:

• containerd.io_1.2.2-3_amd64.deb
• docker-ce-cli_19.03.3~3-0~ubuntu-bionic_amd64.deb
• docker-ce_19.03.3~3-0~ubuntu-bionic_amd64.deb

We are lucky here, because containerd and docker-ce-cli are the only dependencies for docker-ce on the Ubuntu installation which I use (AWS AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*). If your Ubuntu has a minimal set of preinstalled packages and libraries it could be possible, that dpkg will ask for additional dependencies.

As we now have the required packages available, this allows us to write a short piece of Shell script that we can use in the install script as a fallback solution to install Docker CE if the installation using apt with the repositories fails for any reason:

if [ ! `command -v docker` ]; then
  echo "Installing latest Docker version from repositories failed, falling back to manual installation"
  DPKG_URL="https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64"

  CONTAINERD_DEB="containerd.io_1.2.2-3_amd64.deb"
  wget -O $CONTAINERD_DEB $DPKG_URL/$CONTAINERD_DE
  sudo dpkg -i $CONTAINERD_DEB

  DOCKER_CLI_DEB="docker-ce-cli_19.03.3~3-0~ubuntu-bionic_amd64.deb"
  wget -O $DOCKER_CLI_DEB $DPKG_URL/$DOCKER_CLI_DEB
  sudo dpkg -i $DOCKER_CLI_DEB

  DOCKER_CE_DEB="docker-ce_19.03.3~3-0~ubuntu-bionic_amd64.deb"
  wget -O $DOCKER_CE_DEB $DPKG_URL/$DOCKER_CE_DEB
  sudo dpkg -i $DOCKER_CE_DEB
fi

And we’re done!

But please note …

Use apt install whenever possible, or try to use a script that selects the proper packages for your Ubuntu version automatically. The prcoedure I describe in this post is really only a fallback solution to workaround temporary issues with the package repositories. Otherwise a few years or even months down the road you will end up with your stuff mysteriously failing, because you are using an old Docker version which is unsupported by the latest and greatest version of the Kubernetes Engine of your choice that you try to put on top of it.

And that’s all for today.