DO NOT USE THIS REPO - MIGRATED TO GITLAB

AWS Concourse

An AWS based Concourse platform

Built on top of concourse-control-tower

Setting up on first use

In order to install dependencies required by this repo, run make bootstrap

Deploying Concourse

make terraform-init
make terraform-plan
make terraform-apply

Resources

• https://concoursetutorial.com/
• https://github.com/skyscrapers/terraform-concourse

Destroying

Destroy isn’t always possible from Terraform. Manual steps found to be:

• remove prevent destroy in terraform/modules/cloudwatch_log_group/log_group.tf (still valid 11/09/20)
• Delete NAT gateways (still valid 11/09/20)
• The module.vpc.aws_route_table.public resource will remain, due to a peering connection to metrics infrastructure keeping everything alive. (valid 11/09/20))

GitHub Actions user

Details on managing the user credentials can be found here

This user will deactivate its own credentials after every use. In order to activate them again, you must run this job.

AMI testing

Upon a new releae of Concourse, or an update to the Concourse AMI config, we build a new AMI named: untested-dw-al2-concourse-ami using the ami-builder pipeline in Concourse. Upon successful creation of a new untested-dw-al2-concourse-ami resource, we will trigger the validate-concourse-ami job. The only way this job will succeed is if the test results provided by the concourse resource, match the AMI ID provided by the untested-dw-al2-concourse-ami resource, and contain the word SUCCESS.

The concourse resource is provided in an non-standard pattern, which differs entirely from all other test result resources in the ami-builder pipeline. This is mainly due to Concourse being deployed via GitHub Actions and not Concourse, unlike every other consumer of our AMIs.

A new untested-dw-al2-concourse-ami resource, is also consumed by the ami-id-update job. This job uses the new resource to update a secret called ami-ids that is stored in AWS SecretsManager. It’s this secret that is queried when Concourse is deployed by GitHub Actions providing the AMI ID to use when creating/updating the infrastructure. Upon sucessfully updating the secert, the job triggers the AMI build and test workflow. This workflow, when successful, deploys to only management-dev, terminates the existing Concourse EC2s and produces an new GitHub release.
It should be noted that we purposely pin the AMI ID in this job for Management. We do this to ensure we do not unintentionally upgrade the Production Concourse. The aim is to remove this pin when we have built confidence in the test and relase process.

Post-termination, new Concourse nodes will start on the new AMI, allowing the execution of a single pipeline which creates our test results. This is the concourse pipeline, which only exists in management-dev. This consumes the newly created GitHub release, then compares the AMI ID provided by the untested-al2-concourse-ami and the AMI ID on the current running instances of Concourse in management-dev. If every instance matches, then a successful test result is created.

This successful test result will allow the validate-concourse-ami job to now pass, creating the dw-al2-concourse-ami which can be deployed to Management.

Rebooting Worker Nodes/Terminating Instances

Should AWS instances need to be refreshed/replaced then the reboot-worker-nodes pipeline should be utilised which will ‘Land’ the Worker nodes (stop any jobs being sent to them) and then the ASG will replace the instances.