React>> flux-aws-playground>> 返回
项目作者： cloud-native-playgrounds

项目描述 ：
GitOps with Flux, Github Action, Amazon ECR & Amazon EKS
高级语言： Shell
项目地址: git://github.com/cloud-native-playgrounds/flux-aws-playground.git
创建时间： 2020-08-08T07:03:57Z
项目社区：https://github.com/cloud-native-playgrounds/flux-aws-playground
开源协议：MIT License
下载


flux-aws-playground

GitOps with Flux, Github Action, Amazon ECR & Amazon EKS

image

Building and pushing a docker image to Amazon ECR with GitHub Actions

GitHub Actions enables you to create custom software development life cycle (SDLC) workflows directly in your GitHub repository.

Workflows are custom automated processes that you can set up in your repository to build, test, package, release, or deploy any project on GitHub. With workflows you can automate your software development life cycle with a wide range of tools and services.

In this post, you’ll learn how to use a GitHub Actions workflow to build and push a new container image to Amazon ECR upon code change.

You must store workflows in the .github/workflows directory in the root of your repository. The files are in .yml or .yaml format.

Let’s create one called build.yml.

The first part is the name of your workflow. It is used to display on your repository’s actions page.

  1. name: Building and pushing a docker image to Amazon ECR

The second part is on, which is the name of the GitHub event triggering the workflow.

You can provide a single event

  1. on: push

or a list of events 

  1. on: [push, pull_request]

We can also add more configurations. For example, we can specify activity types. The below example shows it triggers the workflow on push or pull request only for the master branch and for the paths under app/**.

  1. on:
  2.   pull_request:
  3.     paths:
  4.     - app/**
  5.     branches:         
  6.     - master
  7.   push:
  8.     paths:
  9.     - app/**
  10.     branches:         
  11.     - master

The next part is env. We’ll setup environment variables to provide configuration option and credentials via Github. 

  1. env:
  2.   AWS_DEFAULT_REGION: ap-southeast-1
  3.   AWS_DEFAULT_OUTPUT: json
  4.   AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
  5.   AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  6.   AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  7.   CONTAINER_IMAGE: example-container:${{ github.sha }}

Go to Github, navigate to Settings in your repository. Click Secrets.

Add three new secrets namely AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY.

image

A workflow run is made up of one or more jobs. They run in parallel by default. Each job runs in an environment specified by runs-on.

A job contains a sequence of tasks called steps. Steps can run commands, run setup tasks, or run an action in your repository, a public repository, or an action published in a Docker registry. 

  1. jobs:
  2.   build-and-push:
  3.     name: Building and pushing image to AWS ECR
  4.     runs-on: ubuntu-latest
  5.     steps:
  7.     - name: Checkout
  8.       uses: actions/checkout@master
  10.     - name: Setup ECR
  11.       run: |
  12.         $( aws ecr get-login --no-include-email )
  14.     - name: Build and tag the image
  15.       run: |
  16.         docker build \
  17.           -t $CONTAINER_IMAGE \
  18.           -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE ./app
  20.     - name: Push
  21.       if: github.ref == 'refs/heads/master'
  22.       run: |
  23.         docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE

Let’s break it out. There is a job called build-and-push. There are four steps running on a virtual environment which is Ubuntu 18.04.

The first step is to check out the master.

  1. - name: Checkout
  2.     uses: actions/checkout@master

Then, we need to setup our Amazon ECR in order to push our image to it. 

  1.     run: |
  2.     $( aws ecr get-login --no-include-email )

The third step is to build and tag the docker image. Notice that we are using the environment variables defined in env. 

  1. - name: Build and tag the image
  2.     run: |
  3.     docker build \
  4.         -t $CONTAINER_IMAGE \
  5.         -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE ./app

The last step is to run docker push to push the image built in the previous step to Amazon ECR.

  1. - name: Push
  2.     if: github.ref == 'refs/heads/master'
  3.     run: |
  4.     docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE

Commit something under app directory and push the changes to master.

Navigate to Actions. You should see a workflow is being processed.

image

You can see the status or check the log for each step.
image

You can see the latest tag name when you expand Build and tag the image.

  1. Successfully built a1ffb1e3955b
  2. Successfully tagged example-container:545385325b99e079cb7ee69d3809efd90cbffba9
  3. Successfully tagged ***.dkr.ecr.ap-southeast-1.amazonaws.com/example-container:545385325b99e079cb7ee69d3809efd90cbffba9

Go to AWS ECR Console, you should see the image there.

That’s it. Here’s the complete build yaml file.

  1. name: Building and pushing a docker image to Amazon ECR
  3. on:
  4.   pull_request:
  5.     paths:
  6.     - app/**
  7.     branches:         
  8.     - master
  9.   push:
  10.     paths:
  11.     - app/**
  12.     branches:         
  13.     - master   
  15. env:
  16.   AWS_DEFAULT_REGION: ap-southeast-1
  17.   AWS_DEFAULT_OUTPUT: json
  18.   AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
  19.   AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  20.   AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  21.   CONTAINER_IMAGE: example-container:${{ github.sha }}
  23. jobs:
  24.   build-and-push:
  25.     name: Building and pushing image to AWS ECR
  26.     runs-on: ubuntu-latest
  27.     steps:
  29.     - name: Checkout
  30.       uses: actions/checkout@master
  32.     - name: Setup ECR
  33.       run: |
  34.         $( aws ecr get-login --no-include-email )
  36.     - name: Build and tag the image
  37.       run: |
  38.         docker build \
  39.           -t $CONTAINER_IMAGE \
  40.           -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE ./app
  42.     - name: Push
  43.       if: github.ref == 'refs/heads/master'
  44.       run: |
  45.         docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$CONTAINER_IMAGE

For more, please check out GitHub Actions Documentation

Deploying Your Application to Amazon EKS with GitHub Actions and Flux

However, if you are using Amazon EKS, you may need to manually update the image URI every time you have a new release. is there a way to automate the whole process that the image URI can be updated automatically? Yes. Here’s the solution for you.

Flux is the operator that makes GitOps happen in your cluster. It ensures that the cluster config matches the one in git and automates your deployments.

Configure your kubectl so that you can connect to an Amazon EKS cluster by running 

  1. export AWS_REGION="ap-southeast-1"
  2. export CLUSTER_NAME="your-cluster-name"
  4. aws eks --region ${AWS_REGION} update-kubeconfig --name ${CLUSTER_NAME}

If you enable load balancer ingress access, make sure that you have the corresponding IAM role. 

  1. aws iam get-role --role-name "AWSServiceRoleForElasticLoadBalancing" || aws iam create-service-linked-role --aws-service-name "elasticloadbalancing.amazonaws.com"

Run your manifest files

  1. kubectl apply -f manifests/deployment.yaml 
  2. kubectl apply -f manifests/service.yaml 
  3. kubectl apply -f manifests/ingress.yaml

A sample deployment can be found here. Make sure you have fluxcd.io/automated: "true" under annotations.

The next step is to run Flux on our EKS cluster. Let’s create a new namespace flux in where flux objects will be installed.

  1. kubectl create ns flux

Install flux objects under flux namespace. By doing so, flux is monitoring the manifests folder for the changes. 

  1. export GHUSER=your-github-user
  2. export GHREPO=your-github-repo
  4. fluxctl install \
  5.     --git-user=${GHUSER} \
  6.     --git-email=${GHUSER}@users.noreply.github.com \
  7.     --git-url=git@github.com:${GHUSER}/${GHREPO} \
  8.     --git-path=manifests \
  9.     --namespace=flux | kubectl apply -f -

You should see the following 

  1. serviceaccount/flux created
  2. clusterrole.rbac.authorization.k8s.io/flux unchanged
  3. clusterrolebinding.rbac.authorization.k8s.io/flux configured
  4. deployment.apps/flux created
  5. secret/flux-git-deploy created
  6. deployment.apps/memcached created
  7. service/memcached created

Let’s verify if they are running or not

  1. kubectl get all -n flux
  1. NAME                             READY   STATUS    RESTARTS   AGE
  2. pod/flux-6449c6bd94-7gz88        1/1     Running   0          5m
  3. pod/memcached-86869f57fd-52cwn   1/1     Running   0          5m
  5. NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
  6. service/memcached   ClusterIP   10.100.152.74   <none>        11211/TCP  5m
  8. NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
  9. deployment.apps/flux        1/1     1            1           5m
  10. deployment.apps/memcached   1/1     1            1           5m
  12. NAME                                   DESIRED   CURRENT   READY   AGE
  13. replicaset.apps/flux-6449c6bd94        1         1         0       5m
  14. replicaset.apps/memcached-86869f57fd   1         1         1       5m

Upon the completion of deployment, the docker image URI in deployment.yaml should be updated. To do so, we need to grand read/write access to the repository with a deploy key so that Flux can be able to write it back every time it deploys.

By running 

  1. fluxctl identity --k8s-fwd-ns flux

You should get a deploy key.

  1. ssh-rsa 
  2. AAAAB3NzaC1yc2EAAAADAQABAAABAQC64WoWesnPneyDqq8ddTAAOKSaLHcu+0ALL8xxtGdnbK2WG99OZ7A9cq24Y9TmSL4gIuXb0HDvwhHsnbkTNsFmVWpO9xS/T3bqhLzhdQwLCGP21ckhRVF7RBv+pK6PnenY4ZjTRkW5h7SxYnunEarj/9E9NlL/JP8tDnb53liDXF4AB1y3Xi/nKwjlgwkGGrSBXGSRij7a6uq2iMlGF/H9MmHn8ct7w/dd/RF6VN4phbNpsVfnBVu1yDgRJTNKznXDOCEEAfflxAFrDWjbAsXwCxvWLNsbP5HtMTf5Ep/Eba7ZAjZ7XnWYLgoXRZHOf+0WYqn1EfsSot5pb01TFeYr

Go to Settings > Deploy keys and click ‘Add deploy key’
image

Enter the title and the key you just generated. Make sure you tick ‘Allow write access’
image

Then we can go back to the console and run the following command to sync Flux and Github.

  1. fluxctl sync --k8s-fwd-ns flux

For the first time, you should see 

  1. Synchronizing with git@github.com:wingkwong/eks-flux-playground
  2. Revision of master to apply is a8e3b45
  3. Waiting for a8e3b45 to be applied ...
  4. Done.

If you make a change and push to master, Github Actions helps to build and push the docker image to Amazon ECR, and Flux helps to deploy the latest image to Amazon EKS.

Go back to the repository, you should see there is a new commit on your deployment.yaml while the change is only updating the image URI. 

  1. Auto-release xxxxxxxxxxxx.dkr.ecr.ap-southeast-1.amazonaws.com/eks-flux…

A Workaround for Syncing and Updating Multiple Repositories with Flux

Flux is the GitOps Kubernetes operator, which is most useful when used as a deployment tool at the end of a Continuous Delivery pipeline. Flux will make sure that your new container images and config changes are propagated to the cluster.

However, at this moment, flux only works with a single git repository containing Kubernetes manifests.

Let’s say you have three applications from three different repositories. If you run fluxctl install for each application on different namespace, and list the controllers with the last namespace you created. 

  1. fluxctl list-controllers --k8s-fwd-ns=app3
  1. WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
  2. default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready   
  3. default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
  4. default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready    automated

If you list the controllers with namespace app1

  1. fluxctl list-controllers --k8s-fwd-ns=app1

There is no workload for it

  1. WORKLOAD                    CONTAINER    IMAGE

Same as app1

  1. fluxctl list-controllers --k8s-fwd-ns=app2

No workload is expected

  1. WORKLOAD                    CONTAINER    IMAGE

Therefore, even you make a commit to repo app1 or app2, it never triggers the controller to sync and update the repo. Your deployment would remain unchanged.

To fix it, run

  1. kubectl edit clusterrolebinding.rbac.authorization.k8s.io/flux

You should see 

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRoleBinding
  3. metadata:
  4.   annotations:
  5.     kubectl.kubernetes.io/last-applied-configuration: |
  6.       {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"name":"flux"},"name":"flux"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"flux"},"subjects":[{"kind":"ServiceAccount","name":"flux","namespace":"app3"}]}
  7.   creationTimestamp: "2020-03-13T16:31:43Z"
  8.   labels:
  9.     name: flux
  10.   name: flux
  11.   resourceVersion: "85027"
  12.   selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/flux
  13.   uid: 202463ba-6548-11ea-a8a2-025c790809a6
  14. roleRef:
  15.   apiGroup: rbac.authorization.k8s.io
  16.   kind: ClusterRole
  17.   name: flux
  18. subjects:
  19. - kind: ServiceAccount
  20.   name: flux
  21.   namespace: app3

Since you create app3 at the end, the cluster role binding config is modified when you run fluxctl install. 

  1. clusterrolebinding.rbac.authorization.k8s.io/flux configured

If you check out flux RBAC template, you can see there is only one subject. 

  1. apiVersion: rbac.authorization.k8s.io/v1beta1
  2. kind: ClusterRoleBinding
  3. metadata:
  4.   name: {{ template "flux.clusterRoleName" . }}
  5.   labels:
  6.     app: {{ template "flux.name" . }}
  7.     chart: {{ template "flux.chart" . }}
  8.     release: {{ .Release.Name }}
  9.     heritage: {{ .Release.Service }}
  10. roleRef:
  11.   apiGroup: rbac.authorization.k8s.io
  12.   kind: ClusterRole
  13.   name: {{ template "flux.clusterRoleName" . }}
  14. subjects:
  15.   - name: {{ template "flux.serviceAccountName" . }}
  16.     namespace: {{ .Release.Namespace | quote }}
  17.     kind: ServiceAccount
  18. {{- end -}}
  19. {{- end -}}

Therefore, to allow three applications at the same time, we need to add the missing two.

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRoleBinding
  3. metadata:
  4.   annotations:
  5.     kubectl.kubernetes.io/last-applied-configuration: |
  6.       {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"name":"flux"},"name":"flux"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"flux"},"subjects":[{"kind":"ServiceAccount","name":"flux","namespace":"app1"}]}
  7.   creationTimestamp: "2020-03-13T16:31:43Z"
  8.   labels:
  9.     name: flux
  10.   name: flux
  11.   resourceVersion: "85027"
  12.   selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/flux
  13.   uid: 202463ba-6548-11ea-a8a2-025c790809a6
  14. roleRef:
  15.   apiGroup: rbac.authorization.k8s.io
  16.   kind: ClusterRole
  17.   name: flux
  18. subjects:
  19. - kind: ServiceAccount
  20.   name: flux
  21.   namespace: app1
  22. - kind: ServiceAccount
  23.   name: flux
  24.   namespace: app2
  25. - kind: ServiceAccount
  26.   name: flux
  27.   namespace: app3

Once you save the file, it will update the config in the background. Now we can verify the result.

  1. fluxctl list-controllers --k8s-fwd-ns=app1
  1. WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
  2. default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    automated    
  3. default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
  4. default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready
  1. fluxctl list-controllers --k8s-fwd-ns=app2
  1. WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
  2. default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    
  3. default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    automated    
  4. default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready
  1. fluxctl list-controllers --k8s-fwd-ns=app3
  1. WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
  2. default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    
  3. default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
  4. default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready    automated

Then when you make a commit to your repo app1, app2 and app3, it should auto release your application and your deployment.yml should be updated by flux with a latest docker image URI.