dummyapp

This is a simple app for working out the flows for automated Docker image
creation and deployment from within CI.

THIS IS CURRENTLY DISABLED: I shut down my Docker Hub bot-integration rather
than pay a minimum of $300/yr for demoing a few open source projects, just to
get team management. The .circleci/config.yml has been deleted.
I vaguely intend to recreate this using an alternative image registry, which
is why this git repository has not been deleted. — 2020-11-24

No warranty. You get to keep all the pieces and shards if it breaks.
There’s a 2-clause BSD license as a formality.

There should be little enough here in terms of “traditional code”, but the
infrastructure of how pieces fit together may be useful to you, after reading
and analysis. If you use any of it, then a word of attribution might be nice
(and will absolve you of the need to honor the formal copyright notice
propagation for build framework).

This Git repo is setup so that pushes automatically trigger builds within
Circle CI, which creates a from-scratch Docker image (using a multi-stage
Dockerfile) and deploys it to both Docker Hub and, for main branch,
to Heroku.

Well, it did, but I’ve shut down the Heroku app, so that bit of logic is
commented out in the .circleci/config.yml file; I’ve left it intact as a
reference.

Terminology

• DinD: Docker-in-Docker
• Controller Image: the image launched by Circle CI to handle the steps from
  .circleci/config.yml; this is not invoked by docker build, but is the
  image where a shell command of docker build will be run
  The Controller Image is not specified outside of the .circleci/ directory.
• Builder Image: the multi-stage Dockerfile creates two images; the first
  is the Builder Image and has a userland, normal tools, a compiler and more.
• Deploy Image: the much smaller image created later in the multi-stage
  Dockerfile, which copies files made in the earlier stages. This is the
  “product” and is what is pushed out.

Setup

We create a Heroku app, enable Go language metrics manually (because using
Docker deploy, not a buildpack), disable git push to the remote (but leave
the remote in place so that the Heroku CLI can auto-determine the deployed
app name), and do a build and deploy with the Heroku tag set.

The build-tag affects both the Docker image tag-name and the content which
is built; for Heroku, it ensures that we compile their metrics push code.

heroku apps:create pt-dummy-app
heroku labs:enable go-language-metrics
heroku labs:enable runtime-heroku-metrics
heroku labs:enable runtime-empty-entrypoint
git config --local --unset remote.heroku.fetch
git config --local remote.heroku.pushurl no_push_because_we_deploy_docker_images
git config --local --bool remote.heroku.skipFetchAll true
# At the time this was done:
#   make BUILD_TAGS=heroku heroku-deploy
# If done today:
env BUILD_TAGS=heroku ./build/build.with-docker.sh
# or equivalently:
./build/build.with-docker.sh env-BUILD_TAGS=heroku

Created repo on Docker Hub through web UI: pennocktech/dummyapp

For a second project, philpennock/poetry (as an example of depending upon
external data) I set up an automated Docker Hub build. That project creates
a data-only Docker image, which we now depend upon at build time. There’s
one COPY --from line in our Dockerfile to edit to remove that.

Created Circle CI project; pushed on branch circle, aborted first build on
master (as the main branch was then called).

NB: the runtime-empty-entrypoint lab came into existence after I first
created this project, but is what lets us skip setting the ENTRYPOINT in the
Dockerfile and just have array-form RUN work correctly.

Authentication

Heroku, Docker Hub

Created a Circle CI org-level Context, heroku-and-dockerhub, added
credentials there for HEROKU_TOKEN, DOCKERHUB_USER, and
DOCKERHUB_PASS.

As to the values:

1. Heroku is bad enough in only having one auth token at a time, unscoped, so
   no ability to trace leaks or constrain actions of the token.
   Run heroku auth:token while signed in, that’s the token to use.
   1. This has changed; you can now use:
      heroku authorizations:create -d 'Circle CI token, created by Fred'
   2. You can adjust the expiration time and the oauth scopes, I’ve not yet
      explored to figure out the minimum scope needed.
2. Docker Hub … defaults to storing your usercode and master password in
   ~/.docker/config.json; you probably want to install
   docker-credential-helper if you haven’t already done so.
3. Docker Hub on v2 has an API for getting tokens, but the offline_token=true
   part of a request is not honored (that I can tell), so you don’t get
   anything usable for passing into another service.
4. So: create a bot account for Docker Hub via normal sign-up; this bot can
   create arbitrary repos of its own, but only under its own account.
   Then in the fully privileged admin account, click Organizations in the
   header, then your organization, then near the top Teams and create a new
   team, dummyapppushers and add the new account to it.
   Then go to the repository page, Collaborators, and add the
   dummyapppushers team with Write access.
   • You can now use the usercode/password for the bot account in
     DOCKERHUB_USER and DOCKERHUB_PASS.

Now update the .circleci/config.yml to reference the context; yes, any build
within the org can request any context, you can’t have admins defining
restricted contexts with some credentials. If you want that, then you’ll need
multiple Circle CI orgs (each with their own billing?).

Google Container Registry

Note: Google have replaced Container Registry with Artifact Registry

Here we can take advantage of a decent permissions model and get a token which
can update just the one image repository.

Ideally it couldn’t delete or remove objects, such that a compromise of the CI
environment does not propagate out to data loss elsewhere; alas, this does not
appear to be possible, as a Docker limitation.

Google:

1. Created a new Google Cloud project (dummyapp-214121), tied to billing account.
2. Enabled Container Registry.
3. Under IAM & admin, in Service accounts, created a service account:
   • Name: circleci-dummyapp-image-builder
   • Role: Storage / Storage Admin (this is temporary)
   • Enable Furnish a new private key (JSON)

Circle CI:

1. Created a Circle CI org-level Context: google-dummyapp
2. Create new environment variable within google-dummyapp
   • Name: GCLOUD_AUTH_ENCODED
   • Value: paste in base64-encoded copy of JSON file generated by Google
     • On macOS, I ran: base64 < ~/Downloads/dummyapp-214121-31db07d29579.json | pbcopy

After doing a run and having the Bucket be created, we can strip away
permissions in Google:

1. Under IAM, remove the permissions for the circleci-dummyapp-image-builder
   account
2. For us.artifacts.dummyapp-214121.appspot.com edit permissions for the
   user, manually grant, for this bucket, access to Storage Admin and
   remove any other permissions for this user.
   Per https://cloud.google.com/storage/docs/access-control/iam-roles on
   storage admin,
   “When applied to an individual bucket, control applies only to the
   specified bucket and objects within the bucket.”

Then re-run the Circle CI push job and confirm can still push.

I’ve tried using a PT: GCR Image Pusher role which drops
storage.buckets.delete and storage.objects.delete (and originally also
storage.objects.update) but it doesn’t work, Docker just fails. This seems
likely to be an infelicity in the operations which Docker tries against the
image repository backend, such that it’s not designed to work with
non-destructive access to blob stores.

Build Dependencies

• We use a pennocktech/ci image for building in Circle; it’s got Go and a
  few other tools.
  GitHub,
  Docker Hub.
• We merge in contents from a data image; at the size we’re at, it’s silly,
  but that’s not necessarily true for larger data sets. I’m using
  philpennock/poetry which is just a couple of Rudyard Kipling poems.
  GitHub,
  Docker Hub.
• A Docker-official golang image, for the Builder Image.
  GitHub,
  Docker Hub.
  • But in Circle, we override this to be the same as the Controller Image.
• The other dependencies are Golang libraries, which we pull in via Golang
  module handling.

All are automated Docker Hub builds as public images from public GitHub repos.
The golang image is from the docker-library GitHub organization, while the
others are from GitHub repos which have names matching the Docker Hub repo
names.

Build Tricks

Use an HTTP proxy during build, and switch the base image:

Create:

DOCKER_http_proxy=http://192.0.2.1:3128/ DOCKER_RUNTIME_BASE_IMAGE=alpine \
  ./build/build.with-docker.sh`

Run:

docker run -it --rm ${imageid} /bin/sh

Before v0.1.0 we defaulted to Heroku bug-compatibility, so had to use
ENTRYPOINT to get around an attempt to invoke /bin/sh for our command,
even when given in array form. From v0.1.0 onwards, we require that Heroku
be told heroku labs:enable runtime-empty-entrypoint which isn’t quite right,
but does at least let us use an array to invoke a command where there is no
/bin/sh inside the container.

Build using a different CI image

This needs too many knobs, we should look at what’s needed to simplify it,
but:

DOCKER_BUILDER_IMAGE=pennocktech/ci:purple \
  EXTRACT_GO_VERSION_FROM_LABEL=com.pennock-tech.versions.go \
  ./build/build.with-docker.sh