Spinning Up an Elastic CI-Pipeline using Jenkins and Kubernetes
With the growth of programming projects and the rise of service oriented architectures, the partitioning of the development process into more specialized tasks becomes inevitable. These tasks will usually be solved by different teams in parallel in order to increase efficiency, leading to faster progress in the development of the source code. However, it creates an additional task of combining the progress of the different teams. Changes introduced by one team might affect the implementation of another, creating conflicts. While small conflicts are usually resolved quickly, a belated identification might lead to a quick growth and let the developers enter “integration hell”. This term describes the point where merging parallel developments of the code and resolving associated conflicts will take up more time than the original code development itself.
To avoid “integration hell”, continuous code integration needs to be done to ensure a timely identification and resolution of conflicts. Therefore, automated continuous integration tools like Jenkins have developed to be an indispensable tool for software development teams. They offer the possibility to automatically build and test integrated code. This can be done periodically or event driven, for example every time a developer pushes new code into a repository. The CI tool then reports back the results to the developers which are then able to resolve possible conflicts immediately.
The development of CI software like Jenkins enables to quickly setup a continuous integration server that can be configured to the specific needs of a project easily. However, hardware requirements for the server might be difficult to estimate upfront. Progress in code development is rarely constant over time: Upcoming software releases will largely increase the amount of code produced and therefore will increase the demands on the continuous integration server. The associated urgency demand that the server will still be capable to perform timely builds and tests of new code. This will lead to higher hardware requirements. On the other hand, a project might hit a period of less development, as developers are concerned with other projects/problems. If this is the case, idling hardware will lead to unnecessary expenses.
The varying demand makes the CI pipeline a perfect target for elastic computing technologies, as for example the Kubernetes container orchestration tool in combination with cloud-computing technologies. This will enable users to automatically spin up servers that build and test newly produced code and terminate them as soon as they are not needed anymore.
In this short tutorial we will create a minimal setup of an elastic CI pipeline using Jenkins within a Kubernetes cluster. This cluster can be used locally, or within a scalable environment like AWS, Azure or Google cloud services. Both Kubernetes and Jenkins offer a large amount of additional setup options for further customization and can be combined with additional tools to complete the development pipeline, for example code repositories like Git and deployment repositories as for example Nexus. The following steps will be performed on AWS-EC2 instances running the Amazon Linux 2 AMI. However, the steps are adaptable to other operating systems (but might need a few adjustments).
The Kubernetes Cluster
Our Kubernetes cluster will consist of one master node and a variable number of worker nodes. As network plugin, we will use Flannel. We will start by installing the Kubernetes tools on all nodes. After this preparation is done, we will initialize the cluster on the master nodes. The worker nodes can then be joined consecutively. The following commands should all be run as the root user (not just sudo, as some commands will not work) on every node.
Assuming the operating system and repositories have been brought up to date via
yum update -y
the setup starts with adding the Kubernetes repository to yum
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
To be able to run the Kubernetes cluster, we need to disable SELinux on our node
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
For yum to accept the Kubernetes repository signature, GnuPG has to be updated
Further, a few iptables settings need to be changed for Flannel to work correctly
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
Now, install the Kubernetes tools
yum install -y docker kubelet kubeadm kubectl --disableexcludes=kubernetes
At this point, it should be checked if the br_netfilter is running
lsmod | grep br_netfilter
If this command does not give any output, br_netfilter needs to be started with the following command
For the services needed by Kubernetes to start automatically when the system boots, the respective services need to be enabled in the system daemon
systemctl enable kubelet.service
systemctl enable docker
systemctl start docker
The Master Node
After running the preparation steps of the last section on every node, it is now time to spin up the cluster by running
kubeadm init --pod-network-cidr=10.244.0.0/16
on the node that will serve as the Kubernetes master. The command will produce an output showing how to join additional nodes to the cluster. The respective join command can be recovered at a later stage by running
kubeadm token create --print-join-command
Note that this command also creates a new token. The tokens expire after 24h and thus need to be created on a regular basis.
The Kubernetes cluster should be administrated by a user different than root, so we need to exit the root shell
and then run
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
to enable the default user to administrate our Kubernetes cluster.
Finally, the Flannel network plugin needs to be started within the cluster
Note that this command will not be runnable as root user within the AWS EC2-Instance, as outbound traffic is disabled for the root user.
Joining Worker Nodes
To join additional EC2 instances as worker nodes, we only need to run the kubeadm join command obtained in the master node setup on each node. It will look something like this:
kubeadm join 172.31.9.52:6443 --token a9og1w.p9ju926ld39aoagb --discovery-token-ca-cert-hash sha256:aba42be8add85f3f6bbd8fa76e765a14ab226038b90fc5a967bb3e7e58d716fa
with IP and tokens specific to every setup. The associated IP is the Kubernetes master IP and will be needed later. Note that the join command tries to access port 6443 on the Kubernetes master node, so that port should be opened for the Kubernetes worker nodes. We can check if everything worked correctly by running
kubectl get nodes
on the Kubernetes master node. It should show two nodes (or more if you decided to join additional worker nodes):
Running the Jenkins Master within Kubernetes
Running a web server like Jenkins within Kubernetes is simple in principle but can get very complicated if all features of Kubernetes should be used. In order to maintain the simplicity of this tutorial, we will restrict ourselves to use a dedicated host within the Kubernetes cluster to launch the Jenkins master pod. This means we will know the IP address of that specific host in order to communicate with the Jenkins master. We start by defining a Kubernetes deployment object on the Kubernetes master node by preparing a configuration file jenkins-master-deployment.yaml. Create a file named jenkins-master-deployment.yaml with the following content:
- name: jenkins
- name: JAVA_OPTS
- name: http-port
- name: jnlp-port
- name: jenkins-home
- name: jenkins-home
applying this deployment object using
kubectl apply -f jenkins-master-deployment.yaml
will launch a pod running a jenkins container on one of the worker nodes. Using
kubectl describe pods
we can identify which node that is and note down the IP address under which it is reachable. This will be the Jenkins master IP. It will be needed later.
Note that this IP is the AWS private IP. If we want to access this node from outside the AWS Cloud, we need to extract the public IP of that node from the AWS console. For me, this is 220.127.116.11 (this will be different for every EC2 instance).
Taking a look at the ports section of the deployment configuration file above, we can see that the container will accept connections through ports 8080 and 50000. The former is the http port for the Jenkins web UI, while the latter will be used for the Java Network Launching Protocol (JNLP), which will allow the Jenkins master to run application on the Jenkins slaves later. For these ports to be accessible within our Kubernetes cluster, we will add two NodePort services, opening these specific ports of the Jenkins master pod for application outside of the Kubernetes cluster. This will be done by creating two service configuration files, jenkins-master-service.yaml
- port: 8080
- port: 50000
and applying them to the cluster
kubectl apply -f jenkins-master-service.yaml
kubectl apply -f jenkins-slave-service.yaml
By doing this, we will map the ports 8080 and 50000 within the Jenkins master pod to two randomly generated ports between 30000 and 32767. We can identify these ports by running
kubectl get services
After doing so, we can access the Jenkins master UI using the IP of the node hosting the port (we identified it earlier using kubectl describe pods) and the NodePort mapped to the Jenkins html port 8080 (here: 32710). Remember to open that port to your client IP.
Within this UI, we can now configure Jenkins as we need it. In this tutorial, we will restrict ourselves to add the Kubernetes cluster as a cloud to run build jobs within Jenkins. To do so, we will visit the “Manage Jenkins” tab and navigate to “Configure System”. At the very bottom of that page, we see the entry “Cloud”. In the associated drop-down menu, we choose Kubernetes, which will spawn a large set of new configuration options. Here, we will only change a few settings:
- Kubernetes URL: This is the (private) IP of the Kubernetes master (obtainable via the AWS console) on port 6443. For this communication to work, the port 6443 needs to be open for other nodes within the cluster.
- Jenkins URL: This is the (private) IP of the Jenkins master (obtainable from kubectl describe pods, see above) on the Jenkins http port (obtainable from kubectl get services). Here, the port 32710 needs to be open for other nodes within the cluster.
- Jenkins tunnel: This is the (private) IP of the Jenkins master (see above) on the Jenkins-jnlp port (obtainable from kubectl get services). Also this port needs to be open for all nodes within the cluster.
In addition to that, we check the box “Disable https certificate check”. We further need to set the correct credentials to allow Jenkins to access the Kubernetes cluster. To do so, we click on “add” in the credentials row, we choose “Jenkins”, then “Kubernetes Service Account” for “kind”. We click on add and then choose the newly added “Secret text” in the credentials drop-down menu. For this service account to have the correct access rights, we need to run the command
kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default
on the Kubernetes master node. Note that this command will give admin rights to this service account, which will be a security issue if your Kubernetes cluster is accessible by others.
Next, we choose “Add Pod Template” at the bottom of the page. This will again generate additional fields. Here, we will change “Usage” to “Use this node as much as possible”. We will further define the raw yaml for the Pod as
- name: jnlp
After saving this configuration, Jenkins is able to start Pods at will in order to run a project. This can be tested by creating a test project. On the Jenkins main page, we click on “new project” and choose freestyle. We configure the project to run a single shell command:
cat /dev/zero > /dev/null
This command will run until it is canceled by user interaction.
In order to test the Kubernetes-Jenkins integration, it makes sense to prohibit builds on the Jenkins master node. We can do this by visiting “Manage Jenkins -> Configure System” and setting the number of build processors to zero. If we now start a build of our test project, the Jenkins server should spawn a Jenkins slave on any available worker node after some time, which will then run the shell command entered before.
With the elastic Jenkins CI pipeline now running smoothly, scaling hardware to its current needs, it is time to integrate it into the development pipeline. A large set of plugins for Jenkins make it easy to connect it to code repositories like GitLab, GitHub, Bitbucket, and more, allowing for customized event driven builds. On the other end, plugins towards repositories as for example Sonatype Nexus or Artifactory allows to automatically deploy successfully built software. Containerizing these applications as well makes up for a consistent, easily maintainable and scalable development platform. We will discuss these further steps in an upcoming blog article, so stay tuned…
We also only scratched the surface of Kubernetes’ potential in this tutorial. The Jenkins master is still running on a dedicated node that we need to know the IP of to access the web UI. Advanced networking options within Kubernetes will help us to address the Jenkins master, regardless of which node it is running on. Persistent volume techniques will allow to keep the server configuration in cases of failover, where the Jenkins master pod might be restarted on a different node. These topics present a more involved deep dive into the complex mechanisms of Kubernetes and Docker, which will in large parts be covered in our upcoming seminar series „Operationalisierung von AI-Workloads durch Container-basierte Applikationen„.