Jenkins builds inside Kubernetes on EC2/EKS and ECS (Fargate)
This repo contains code for demo purposes only, it contains hardcoded values and no security hardening, so it never should be used in production
aws cli
aws-iam-authenticator
ssm-run
kops
kubectl
kubedecode
helm
kops create cluster --zones eu-west-1a demo1.demo.kagarlickij.com --state s3://kag-kops-state --ssh-public-key ~/.ssh/kops.pub --yes
kops validate cluster demo1.demo.kagarlickij.com --state s3://kag-kops-state
kubectl cluster-info
kubectl apply --filename=k8s-tiller-service-account.yaml
helm init --service-account tiller
helm version
helm install stable/jenkins --name jenkins-master --values helm-jenkins-master-values.yaml
kubectl get pods --watch
printf $(kubectl get secret --namespace default jenkins-master -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
JENKINS_URL=http://$(kubectl get svc --namespace default jenkins-master --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}"):8080/ && python -m webbrowser $JENKINS_URL
Open Jenkins URL and go to Manage Jenkins > Manage Plugins > Updates,
..and select all available updates, than install and restart Jenkins.
Open Jenkins URL and go to Manage Jenkins > Configure system > Cloud > Kubernetes
Open Jenkins URL and go to Manage Jenkins > Configure system > Cloud > Kubernetes > Kubernetes Pod Template,
..and change label from jenkins-master-jenkins-slave
to k8s-agent
pipeline {
agent {
node {
label 'k8s-agent'
}
}
stages {
stage ('build') {
steps {
echo "Hello World!"
}
}
}
}
kubectl get pods --watch
aws cloudformation create-stack --stack-name demo4p1 --template-body file://aws-eks-cluster.yaml --capabilities CAPABILITY_NAMED_IAM
aws cloudformation wait stack-create-complete --stack-name demo4p1
aws cloudformation describe-stacks --stack-name demo4p1 | jq --raw-output '.Stacks | .[] | .StackStatus'
aws eks update-kubeconfig --name demo4
kubectl cluster-info
aws cloudformation create-stack --stack-name demo4p2 --template-body file://aws-eks-nodes.yaml --capabilities CAPABILITY_NAMED_IAM
aws cloudformation wait stack-create-complete --stack-name demo4p2
aws cloudformation describe-stacks --stack-name demo4p2 | jq --raw-output '.Stacks | .[] | .StackStatus'
kubectl apply --filename=k8s-aws-auth-config.yaml
kubectl get nodes --watch
aws cloudformation create-stack --stack-name demo5 --template-body file://aws-ec2-jenkins.yaml --capabilities CAPABILITY_NAMED_IAM
aws cloudformation wait stack-create-complete --stack-name demo5
aws cloudformation describe-stacks --stack-name demo5 | jq --raw-output '.Stacks | .[] | .StackStatus'
ssm-run "cat /var/lib/jenkins/secrets/initialAdminPassword" $(aws cloudformation describe-stacks --stack-name demo5 | jq --raw-output '.Stacks | .[] | .Outputs | .[] | select(.OutputKey == "InstanceId").OutputValue')
JENKINS_URL=http://$(aws cloudformation describe-stacks --stack-name demo5 --region eu-west-1 | jq --raw-output '.Stacks | .[] | .Outputs | .[] | select(.OutputKey == "InstancePublicIp").OutputValue'):8080/ && python -m webbrowser $JENKINS_URL
kubectl config use-context demo1.demo.kagarlickij.com
Open Jenkins URL and go to Manage Jenkins > Manage Plugins > Available > Filter > Kubernetes
..and install it with Jenkins restart
Open Jenkins URL and go to Manage Jenkins > Configure Global Security > Agents > TCP port for JNLP agents > Fixed > 50000
Username: admin
Password: value from
kops get secrets kube -oplaintext --state s3://kag-kops-state
Description: kops-basic-auth
kubectl get secrets
kubedecode default-token-$$$$$
Open Jenkins URL and go to Manage Jenkins > Configure system > Cloud > Add a new cloud > Kubernetes and enter:
Kubernetes URL (kubectl cluster-info
)
Kubernetes server certificate key (from previous step)
Check ‘Disable https certificate check’ option
Use default
for Kubernetes Namespace
Click ‘Text Connection’ button
Enter Jenkins URL
Enter Jenkins tunnel
Name: ‘kops-pod’
Namespace: ‘default’
Labels: ‘kops-agent’
Usage: Use this node as much as possible
Name: jnlp-slave
Docker image: jenkins/jnlp-slave
pipeline {
agent {
node {
label 'kops-agent'
}
}
stages {
stage ('build') {
steps {
echo "Hello World!"
}
}
}
}
kubectl get pods --watch
kubectl config use-context arn:aws:eks:eu-west-1:709237651222:cluster/demo4
kubectl apply --filename=k8s-jenkins-service-account.yaml
kubectl get secrets
Secret: value from
kubedecode jenkins-token-$$$$$
Description: eks-token
Open Jenkins URL and go to Manage Jenkins > Configure system > Cloud > Add a new cloud > Kubernetes and enter:
Kubernetes URL (kubectl cluster-info
)
Kubernetes server certificate key (from previous step)
Check ‘Disable https certificate check’ option
Use default
for Kubernetes Namespace
Click ‘Text Connection’ button
Enter Jenkins URL
Enter Jenkins tunnel
Name: ‘eks-pod’
Namespace: ‘default’
Labels: ‘eks-agent’
Usage: Use this node as much as possible
Name: jnlp-slave
Docker image: jenkins/jnlp-slave
pipeline {
agent {
node {
label 'eks-agent'
}
}
stages {
stage ('build') {
steps {
echo "Hello World!"
}
}
}
}
kubectl get pods --watch
aws cloudformation create-stack --stack-name demo8 --template-body file://aws-ecs-cluster.yaml
aws cloudformation wait stack-create-complete --stack-name demo8
aws cloudformation describe-stacks --stack-name demo8 | jq --raw-output '.Stacks | .[] | .StackStatus'
aws ecs describe-clusters --cluster fargate | jq
Open Jenkins URL and go to Manage Jenkins > Manage Plugins > Available > Filter > Amazon Elastic Container Service
..and install it with Jenkins restart
Open Jenkins URL and go to Manage Jenkins > Configure system > Cloud > Amazon EC2 Container Service Cloud and enter:
Name: fargate
Amazon ECS Region Name: eu-west-1
ECS Cluster: arnecs
:cluster/fargate
Click on Advanced tab
Enter Tunnel connection through
Enter Alternative Jenkins URL
Container Cleanup Timeout: 10
Label: fargate-agent
Template name: fargate
Launch type: FARGATE
Soft Memory Reservation: 1024
CPU units: 512
Subnets: subnet-3c1bbc66, subnet-46bbde20, subnet-860773ce
Security Groups: sg-e0b51a91
Enable ‘Assign Public Ip’ option
pipeline {
agent {
node {
label 'fargate-agent'
}
}
stages {
stage ('build') {
steps {
echo "Hello World!"
}
}
}
}
aws ecs list-tasks --cluster fargate | jq
kops delete cluster --name=demo1.demo.kagarlickij.com --yes --state s3://kag-kops-state
aws cloudformation delete-stack --stack-name demo4p1
aws cloudformation delete-stack --stack-name demo4p2
aws cloudformation delete-stack --stack-name demo5
aws cloudformation delete-stack --stack-name demo8