Dibctl

Dibctl is a software for image lifecycle management.
It helps to build, test, upload and delete images in multipie
openstack regions in a consitent way.

Internally it uses diskimage-builder to build images and pytest
with testinfra to test them.

Dibctl uses configuration files to describe how to build image,
which name it should have after upload, what properties (if any)
should be set for a given image. Image configuration file also
provide list of tests for each image, plus name of environment
where tests should happen.

Testing happens under directives from test.yaml. It contains
information how to run a test instance: region authorization URL,
credentials, flavor, network list, availability zone, security
groups and other nova parameters.

Third configuration file (upload.yaml) provides information
for uploading that image into Openstack installations.

Image Lifecycle

Dibctl assumes following image lifecycle:

• Image developed and debugged by operator using command line
  interface.
• Resulting configuration is used in CI server to build image
• Newly build image is then tested
• If tests passed image is uploaded to one or more regions
• During upload process older copies of image are marked as obsolete
• Obosolete images which are no longer in use deleted

Steps “build, test, upload, obsolete, delete” repeated on
periodic basis.

Testing frameworks

Dibctl provides few frameworks for image testing.
Each of those frameworks provides tests with full information
about image,it properties, created instance (it flavor, network
settings, credentials to access instance SSH, etc).
Test frameworks:

• ‘shell’: each test is a simple shell script, which is executed
  outside test VM. It is provided with all information on how
  to connect to the test VM.
• ‘pytest’ - tests are implemented by means of py.test, with optional
  support for testinfra (python library for server verification, similar to
  ServerSpec). Dibctl provides vast set of fixture containing all available
  information about the image and the instance, plus few handy operations
  (wait_for_port), and a direct access to ‘nova’ object of openstack
  client for testing instance reactions on nova operations
  (hard reboot, rebuild, etc).
• ‘ssh’: each test is a simple shell script, which is executed
  inside a guest machine (not yet implemented)
• ‘image’: passive check of the image content (not implemented yet)

Dibctl comes with some generic tests (which should be applicable
to any image of any provider).

Few examples:

• Does instance resize rootfs up to a flavor size at a first boot?
• Does it receive settings and configure IP addresses on all attached interfaces?
• Does DNS resolver set up properly?
• Does hostname match the name of the instance?
• Does instance work after reboot?
• Can user install nginx and get access to http port 80?

Workflow in details

After operators describes configurations, following workflow excepted:

• build: create new image py means of diskimage-builder
• test: new instance is spawned from image under test, and corresponding
  tests are called. If they all report success, images passes the test.
• upload: If test was successful, the image is uploaded to to one or more
  installations of Openstack.
• Older copies of images marked as obsolete and removed (after they become
  unsed - see description below).

That process may be repeated on regular basis via cron or CI server (Jenkins?).
Comprehensive testing assures that image that passed the test may be uploaded
safely in automated manner.

At the same time dibctl may be used by operators from command line and it
provides some means of interfactive debug of faulty images:

• --shell option allow to open shell to instance which failed some tests
• --keep-failed-instance allow to keep instance alive (normally it should
  be deleted at the end of the test)
• shell command allow open shell without running any tests.

Motivational introduction

Diskimage-builder solved most of the issues around process of building images.
It reduces all complexity of images to set of elements and well-defined rules
of the build process.
Nevertheless, there are many problems outside of the diskimage-builder scope:

• Integration testing
• Linking together diskimage-builder-related information (environment variables,
  command line options) and glance-related information (image name, properties
  credentials for upload, etc)
• Unified upload to one or more regions
• Recycling of older image copies

Let’s look to all those sages. Firstly, one need to set up:

• environment variables for diskimage-builder
• few very inconsistent and cryptic variables for used elements
• some command line arguments for diksimage-builder
• list of elements used to build an image

Resulting command line is a ‘golden artifact’ - you need to keep it
somewhere.

After the image was build, one can upload it to Glance. One need
to provide few more pieces of information:

• Image name
• Credentials for Glance
• Additional meta needed to set up on image (hw_property, etc)

This adds few more lines to the ‘golden artifact’.

Normally one wants to test images before uploading. Each image should, at least,
be able to boot and accept ssh key.

One may create a simple script to boot and test it:

This adds few more lines to ‘golden artifact’:

• credential to spawn instance
• flavor id
• nics net-id
• security group name

If one have more than one image with more than one configuration
for testing, this brings up complexity even higher, as many of those
values become variables for reusable part of the script.
If tests failed one should not leave broken and forgotten instances,
ssh keypairs and test images. So there should be garbage collection
code or some kind of ‘finally’ clause in the script.

After upload one want to remove older copies of the same image.
That adds even more lines to the script, bringing it to the scale of
a normal application.

Dibctl was created as evolution of such script, which at certain
point become unmanageable. Newer version was written with better understanding
of the process, without cutting corners.

Outside of its main goal, dibctl gives one more nice feature: image-transfer,
which can copy image from one glance to another while preserving every propery,
ownership and share information (tenant-name based).

Key concepts

Dibctl uses following concepts:

• label (for image and environment) - internal ‘name’ for a given image or environment to
  use in command line and in cross-reference fields.
  All entries in dibctl configuration have label, or ‘namelabel’.
• image: set of attirbutes to build and test actual image.
• test instance: Instance which is used for image testing. Test instance is created
  at test time and removed afterwards. Dibctl creates custom SSH key for each test.
• test image: image which is uploaded for testing. Dibctl uses separate upload stage
  for testing and actual ‘upload to procution’. Test images normally uploaded to specific
  project and are not public. Production images are normally public (or upload to
  selected tenant and shared with specific tenants). Test image is removed after test
  (succesfull or not). This can be changed by —keep-failed-image option.
• test environment: set of attributes and variables describing how upload test image and
  how to boot test instance. Every image references to test environement by it’s label.
  They are listed in tests.yaml
• upload environments: Those describe how and where upload images for production use.
  Uploaded images are subjected to optional ‘obsoletion’ stage (see Obsoletion part below),
  which happens automatically every time image is uploaded or manually.

Image lifecycle

Any of those stages is optional an may be ommited.

Build stage

It uses information from corresponding entry in images.yaml to execute diskimage-builder.
‘filename’ option specify target filename.
dibctl translates diskimage-builder exit codes
(it exits with same exit code as diskimage-builder).

Test stage

At this stage image should be build. Dibctl uses information from entry in images.yaml
to find image file for testing (‘filename’), and than uses environment_name to
find correspoinding test environment in tests.yaml. Then it upload image, creates new ssh key,
spawns instance accoring to test environments settings, and executing tests
in tests_list section of image configuration. Those tests recieve information about
instance (IP addresses, network settings, hostname, flavor, etc), perform instance validation.
If all tests passes successfully, dibctl sets exit code to 0, otherwise error returned.
Regardless of the test results all test-time objects in Openstack are removed: test image,
test instance, ssh key. User may use --keep-failed-image and --keep-failed-instance to
keep them for closer investigation.

By using —shell dibctl may be instructed to open ssh shell to test machine when tests failed.
After that shell is closed, instance (and all other pieces of test) are removed.
If operator wants to keep instance from been removed after shell is closed, he (she) may
use ‘exit 42’ command. Dibctl will honor exit code 42 by not removing instance.

Upload stage

At the upload stage image is uploaded to specified installation with specific settings
for publication. All image-specific things (properties, tags, etc) from images.yaml
are used during this stage, as well, as settings from upload.yaml (See variable ordering
to see override rules).

After upload done, it triggers obosoletion stage if obsoletion is stated in
upload configuration.

Obsolete stage

Obsolete image: If image is in the same tenant and have same glance name as freshly uploaded,
it is obsolete. Obsoleted images recieve specific rename pattern (usually adds ‘Obsolete ‘ before
name), and specific set of properties.

Obsoletion may be performed manually by ‘obsolete’ command.

Rotation stage

If obsolete image is no longer used by any instances in installation it’s called ‘unused obsoleted
image’ and may be safely removed. That is done by ‘rotate’ command.

Please note, rotation requires administrative privilege (dibctl needs to see all instances
in the region). Normally it’s performed periodically by administrator itself, without
delegating this job to CI/cron.

Installation

You need to have following packages installed:

• diskimage-builder
• novaclient
• glanceclient
• keystoneauth1

Important notice: at this moment diskimage-builder package in Debian & Ubuntu is very,
very old (1.0). You need to upgrade it al least to 1.9 to have working images.

Please use pip version or rebuild package if you have own CI/buildfarm.

If you want to use python-based test you will also need:

• pytest
• pytest-timeout
• pytest-testinfra (it’s new library and it may not be in your distro yet, use pip
  or build your onw package).

Note: Due to bug [1] Ubuntu Xenial’s bug, you may recieve pytest warnings when
using pytest runner. Ubuntu’s python-pytest-timeout package is obsolete.
Those warnings are harmless, but if you want to get rid of them, upgrade
python-pytest-timeout package to the newer one.

[1] https://bugs.launchpad.net/ubuntu/+source/python-pytest-timeout/+bug/1676301

TODO: set up ppa with dependencies

Configuration

Every configration file is consists of independent entities. When dibctl scans
for configuration it works with all possible configuration file locations
in a spefic order. Every found configuration file is merged into
common configuration (separately for each type). If configuration file
with higher priority has entity with same label as in files with less
priority, it is replaced with newer entity, and warning is logged to
console.

There are 3 places where configuration are searched: local directory,
‘dibctl’ calalog in current directory and ‘/etc/dibctl’.

There are 2 types of configuration files: monolythic and ‘conf.d’ style.
Files from ‘.d’ catalogs are loaded in alphabetical order.
All together that gives 6 different configuration file locations for
earch configuration type.

There are three different types of configuration files: images (images.yaml),
test environments (test.yaml) and uploads (upload.yaml)

Lookup order for image configuration

(sorting is from the least priority to the highest priority)

• /etc/dibctl/images.yaml
• /etc/dibctl/images.d/*.yaml
• ./dibctl/images.yaml
• ./dibctl/images.d/*.yaml
• ./images.yaml
• ./images.d/*.yaml

Lookup order for test environment configuration

(sorting is from the least priority to the highest priority)

• /etc/dibctl/test.yaml
• /etc/dibctl/test.d/*.yaml
• ./dibctl/test.yaml
• ./dibctl/test.d/*.yaml
• ./test.yaml
• ./test.d/*.yaml

Lookup order for upload environment configuration

(sorting is from the least priority to the highest priority)

• /etc/dibctl/upload.yaml
• /etc/dibctl/upload.d/*.yaml
• ./dibctl/upload.yaml
• ./dibctl/upload.d/*.yaml
• ./upload.yaml
• ./upload.d/*.yaml

If pytest-based tests are in use, than tox.ini and other pytest-related
configuration files may influence tests discovery.

images.yaml

This file describes how to build image, which name and properties it
should have in Glance during upload, which test_environment to use
to test this image, which tests should be ran during test stage.

test.yaml

This file describes test environments. They may be referenced by
images in image.yaml, or forced to be used during test from
command line.
It contains all openstack information for test
purposes:

• Where to upload image for test (OS credentials)
  (images are upload for test independently from
  actual ‘upload’ stage)
• Which flavor, network(s), etc to use

upload.yaml

This file contains configuration for upload.

Variable override ordering

When dibctl performs tests or uploads it combines
information from images.yaml and correspoding
environment config (test.yaml or upload.yaml).

Each of that file may contains glance section.
Normally all image-specific variables should
be kept in the images.yaml config file,
but under certain circumstances it may be desirable
to override/change them for a given location.
Most noticable and common are timeout settings
(for remote/slow regions) and glance_endpoint
to help deal with complicated intra-extra-net
endpoints.

Special priority rules for glance section:

• api_version - environment have priority over image
• upload_timeout - used a max of all available values
• properties - merged. If there are same properties in
  the images config and in environement config,
  environments have priority over image.
• tags - merged (this is a simple list)
• endpoint - environment have priority over image
• protected - max of all values (True > False)
• min_disk - max of all values
• min_ram - max of all values
• all timeout values: max of all values

For all other variables image has priority over environment.

There are examples of configuration files in
docs/example_configs/ folder of this repository.

Writing a test

General conceptions for good py.test test are applied:

• each test checks only one thing
• use assert/raise to signal errors

• dibctl build imagelabel [-o filename] [--images-config images.yaml]
  Build given image

• dibctl test imagelabel [-i filename] [--images-config images.yaml] [--test-environments-config env.yaml] [--upload-only] [--use-existing-image uuid] [--force-test-env env-name]

Upload image to test tenant and spawn instance, run tests against this instance. Return -1 if test failed, or 0 if tests passed.
If no filename supplied, ‘filename’ form images.yaml is used. os_tenant_name, os_password, os_auth_url, os_username may
be overrided via environment variables, and may be ommited in config

• dibctl upload imagelabel [-i filename] [--images-config images.yaml] [--no-obsolete]

Upload image to a given environment with properties and name from images.yaml
Old images with same name (in the same tenant) would be renamed to ‘Obsolete (oldname)’ and marked with property.

• dibctl mark-obsolete uuid [uuid, ...]
  Obsolete given image name (rename and mark it with property)

• dibctl rotate ["Image Name 1" "Image Name 2" ...]

Rotate (remove) unused obsolete images with given name (if no name given, all unused obsolete images are processed)
Requires administrative permissions to find if image is unused or not.

Dibctl uses environment variables for Openstack credentials (except for the tests where it uses credentials from test-environments.yaml).
It uses standard names for environment variables:

OS_USERNAME
OS_PASSWORD
OS_TENANT_NAME
OS_AUTH_URL

For test command dibctl prioritize environment configuration with exception of OS_PASSWORD,
which have higher priority over configuration file.

Exit codes

When everything went well, exit code is 0.
If there is an error with configuration file, or connection error, exit code is -1
If diskimage-builder failed, exit code is the same as disk-image-create’s exit code
If there were test or upload error, exit code is positive (not implemented yet)

Timeouts

For most of operations Dibctl have a time limit. Those timeouts have reasonable
default values, but it’s possible to override them. When the same timeout is
specified in two configuration files, dibctl uses maximal value.
For example, if image config in glance section has upload_timeout: 300, and
environment has upload_timeout: 3600, Dibctl will wait up to 3600 seconds during
image upload.

All time-related settings for Dibctl are measured in seconds.

List of timeout settings (default value is provided in braces):

• glance.upload_timeout (360 seconds). Limit how long image may be uploaded.
  Applied to test, upload and transfer commands.
  All operations stops if this timeout was triggered. Glance image would
  be left half-uploaded, and, hopefully, would be cleaned by glance.
  May be specified in images.yaml, test.yaml and upload.yaml configs.
• tests.wait_for_port (61 seconds). Limit how long to wait instance between
  become ACTIVE and answering on a wait_for_port port. Applied to test
  command only.
  May be specified in images.yaml and test.yaml configs.
• nova.create_timeout (10 seconds): Limit how long nova may answer to
  instance creation request. It does not include time for transfer from
  ‘created’ to ‘BUILD’ and ‘ACTIVE’ states, only the actual ‘create’ command.
  Applied only to test command.
  May be specified in images.yaml and test.yaml configs.
  If this timeout was triggered, uploaded image would be removed, keypair
  will be removed and instance would be left ‘as is’.
  All further operations would be stopped.
• nova.active_timeout (360 seconds). Limit how long dibctl will wait
  for instance after it was created before it become ‘ACTIVE’ (or fall
  into ‘ERROR’ state).
  May be specified in images.yaml, test.yaml and upload.yaml configs
  Applied to test command only.
  If this timeout was triggered tests would not be called.
  Further operations are dependent on command line options:
  image will be removed unless there are —keep-failed-image option.
  instance will be removed unless there are —keep-failed-instance option.
  keypair will be removed anyway, but private key in filesystem would be
  kept in instance was kept.
• nova.keypair_timeout (10 seconds). Timeout for keypair operations.
  Normally you wouldn’t need to change this.
  May be specified in images.yaml, test.yaml
  Applicable to test command only.
  If this timeout was trigged, image would be removed, keypair left
  as is, all further operations would be stopped.
• cleanup_timeout - not yet implemented.
  Applied to test command only
  May be specified in images.yaml, test.yaml

• Each element in tests.tests_list has own timeout value, which
  is limiting time for all tests gathered under given path combined.
  For example, if you have test like this:

  tests:
    tests_list:
      - pytest: /foobar.py
        timeout: 300

  Than if foobar.py contains 3 tests each finishing in 120 seconds,
  than timeout would be triggered at the third test, at 300th second.
  May be specified only images.yaml
  Applied only to test command.
  If this timeout was triggered, behavior is dependent on
  —keep-failed-image, —keep-failed-instance
  and —continue-on-fail option. Regardless of those options,
  rest of tests under given line (which triggered timeout)
  would be skipped (This is due to the test process terminated abruptly).
  After failure or successful operation there is a cleanup stage.
  Each operation in this stage uses same cleanup_timeout value.
  If one operation is triggered timeout, dibctl will continue to perform
  next operation in cleanup cycle.

  If by any reason you want to stop dibctl to perform cleanup cycle
  after you ran it, you may at your choice:

  • use kill -9 to it
  • press Ctrl-Z and than kill -9 %1 (everywhere except for opened shell)
  • use exit 42 in the opened shell to disable cleanup sequence (works
    only inside shell due to obvious reasons).

Pressing Ctrl-C during all operations will threated as failure and would
cause clean sequence accordingly to the command line settings.

Timeout sequence:

• upload_image: wait up to glance.upload_timeout
• create test keypair: wait up to nova.keypair_timeout
• spawn test instance: wait up to nova.create_timeout
• wait for instance to become ACTIVE: wait up to nova.active_timeout
• wait for instance to answer port: wait up to tests.wait_for_ports
• each entry from tests_lists will be capped at it own timeout value.

After tests (or after failure, or after user exited a shell from instance
and that instance should be cleaned up due to command line settings)
Eeach clean operation will be capped up to cleanup_timeout value.
If timeout happens

Individual tests may apply own time limits. For shell tests
it’s normally done with ‘timeout’ command, for pytest-based
tests it is done with pytest.mark.timeout decorator.

Example for the test below has timeout value 15:

@pytest.mark.timeout(timeout=15)
def test_hostname(ssh_backend, instance):
    assert ssh_backend.SystemInfo.hostname == instance.name.lower()

Getting started

There are two distinct workflows you may follow:

• git-based CI/CD style (git style)
• system-wide configuration style (system config style)

git style

In this mode all configuration files, tests and diskimage builder
elements are stored in a single git repository.

This example will operates on following assumptions:

• You already has installed dibctl system-wide (via pip install or
  through apt-get)
• You repository will be called images.
• You have one images to build: Debian Jessie
• You have two installations to upload images: cloud-us and cloud-uk

Preparation:

• mkdir images
• cd images
• git init
• mkdir dibctl
• mkdir tests

Those commands will initialize new repository images with
catalogues dibctl (we will store dibctl configuration files here),
‘tests’ (where we’ll store our tests there).

Configuration file for images

We will create it at dibctl/images.yaml:

debian:
  filename: debian.img.qcow2
  glance:
    name: "Automated Debian Image"
  dib:
    environment_variables:
      DIB_RELEASE: jessie
    elements:
      - debian
      - vm
  tests:
    ssh:
      username: debian
    wait_for_port: 22
    tests_list:
      - shell: tests/reboot.sh
      - pytest: tests/generic.py

Example above is a very short configuration file, there are
many options to tweak, check docs/example_configs to see
all possible options.

Comments:

We use system-wide elements for diskimage-builder
here. If we want to use custom elements, we need to specify
environment variable ELEMENTS_PATH in section ‘dib’:

image-name
  dib:
    environment_variables:
      ELEMENTS_PATH: 'elements'
    elements:
      my-custom-element

In this case path may be relative (to the working catalog
of dibctl) or absolute path.

ssh section in tests specify settings for ssh - in our
case it’s just a username.

wait_for_port specify which port to wait after instance
was created. Dibctl will wait (up to timeout, default is 61 seconds,
but can be overriden with port_wait_timeout option) for
given TCP port to start accepting connections (simple SYN/ACK test
without actual transmission of anything).

tests_list define which tests we want to run. Those tests will
be created in a section below. We are using two test frameworks:

• shell - shell scripts, which are executed outside of instnace
  and have special set of variables providing information about
  instance to test.

• pytest - execure pytest tests outside of of instance but
  have special ssh_backend fixture which allows connection to
  ‘inside’ of guests.

Configuration file for test environment

system-config