Docker-Registry-Util

Search and cleanup on Docker Registry v2.

Project: https://github.com/merll/docker-registry-util

Overview

This project allows queries on existing repositories and tags of a Docker Registry (v2). It comes with a command line
utility dregutil, but can also be used as a lightweight API. Besides gathering information images can also be
deleted from a private Docker Registry, allowing for garbage collection.

Background

It is a good idea to clean up a private Docker registry every now and then. For example, images that have only been
generated for testing purposes and never been sent to production could be deleted for freeing up storage space.

Since version 2.4.x, the Docker Registry comes with a
garbage collector. According to its
documentation, it deletes all blobs not referred to by any image manifest, freeing up space.

The storage of the Docker registry itself is similar to the Docker host image collection: The Docker Registry works on
content-based storage, with the possibility for adding tags. If we take for example an image that has a certain SHA256
digest sha256:00... and two tags: 1.0.0 and latest. It is possible to address (e.g. fetch) the image using
both the digest and either tag. If we upload an image with digest sha256:11... and tags 2.0.0 and latest,
the latest tag will be removed from the sha256:00... image.

Other than the Docker API, which tells all the tags of a certain image id, the Registry API only lists repositories
and tags and shows the associated digest. Technically an identical digest can even be shared by multiple repositories.
If we want to remove an image, we want to make sure that we do not destroy unknown repositories and tags.

This utility presents this relation in both ways. When a tag is selected for deletion, its related digests are
reverse-checked for an association with any other repository and tag.

The entire registry tags and digests are read, and then stored locally in order to speed up further processes.
Therefore, the aforementioned reverse-check may take place on outdated information. It is recommended to update the
cache frequently using the --refresh argument and - like the Docker Registry garbage collection - only perform any
change operations under low or no traffic on the registry.

Registry configuration

In order for this tool to work, the Docker Registry needs to be configured for allowing deletion. In the configuration
file, add the following:

delete:
  enabled: true

Besides that, you need to need to set the environment variable REGISTRY_STORAGE_DELETE_ENABLED to true when
creating the container. This is not documented in the
Registry Configuration Reference but was required last time I checked
and subject to some issue reports (e.g. #989 and
#1573).

Installation

This library is implemented in Python 3. After downloading and installing Python,
installation is simply done via pip:

pip install docker-registry-util

for installing the latest release or

pip install git+https://github.com/merll/docker-registry-util.git

for installing the latest development version.

Getting started

The library first needs to know how to connect to your registry. The following can be set via the command line or
through environment variables:

The REGISTRY / -reg specification is required. If REGISTRY_USER / -u is not specified, the tool attempts to
look up authentication information from the local Docker CLI configuration (~/.docker/config.json).

With this basic configuration, you can query the registry contents via the command line, e.g.

dregutil list-repo-names

lists all the repository names from the registry.

dregutil list-tag-names

lists all available repositories and tags.

On the first start, the tool will query all digests and tags from the registry. In order to speed up operations, this
information is stored in a local cache file named according to the registry name. For example, setting the registry
to registry.example.com stores the cache in registry_example_com_cache.json. If you have done any upload operations
or deletes outside of this tool, you can force a refresh using the --refresh command line argument; you can also
relocate the cache file using the -c argument or deactivate the cache entirely setting -c None.

Queries

Queries will list the digests that are used by particular repositories or tags.

Repositories can be queried by exact name:

dregutil query-repos -r my-repo

Tags can be listed using exact names or version selectors.

dregutil query-tags -r my-repo -t latest

Alternatively, you can query versions by prepending an operator <, >, <=, >=, or == to a version number.
Make sure to escape the > and < for the shell you are using.

dregutil query-tags -r my-repo -t \<1.4

Queries by regular expressions are also possible.

dregutil query-tags -r my-repo -re 1\\.*

For excluding a tag or version match, use the -x argument. For a RegEx-based exclusion, use -xre.

Deletion

Deleting digests from the registry is possible using the same syntax as for queries, but using remove-repos or
remove-tags.

dregutil remove-repos -r my-repo

removes all digests that belong to the repository my-repo from the registry.

dregutil remove-tags -r my-repo -t \<1.4

marks all tags of my-repo as deleted that carry a version number lower than 1.4.

Partial vs. complete match

As image digests may belong to multiple tags, there is a possibility that you might select tags for deletion that are
shared with other images, but that you had not intended to remove. Therefore the default behavior is to reverse-check
selected digests against your original selection. For example

dregutil remove-tags -r my-repo -t \<1.4

will by erase all digests with a version number lower than 1.4, unless they belong any other tag or repository. If
one of the images is also tagged as my-repo/one, it is not removed, unless

dregutil remove-tags -r my-repo -t \<1.4 one

is specified.

This behavior can be changed generally using the --no-match-all-tags command line option. You can also raise an error
on the event of unexpected intersections using the --raise-intersecting-tag argument.

Additional environment variables

The following settings from the command line can also be set in an environment variable.

Further information

A complete reference to available commands and arguments is available via dregutil --help, or
dregutil [command] --help for details about single command.