NGINX Ingress Controller with HAProxy for k8s cluster

Originally posted at https://mrturkmen.com/setup-ingress-controller

In recent post, which is Setup Highly Available Kubernetes Cluster with HAProxy a highly available Kubernetes cluster is created. However, once I started to dig in and deploy some stuff to cluster, I realized that I am not able to connect any deployed application or services. For instance, when an web application is deployed using HAProxy load balancer (endpoint), and check from kubectl (on client side), its status is running. However, that application could not be reached from outside world although I re-patch an external IP address by following command

After some searching and reading, I realized that worker nodes require their own ingress controllers in order to forward traffic between them in case of load. I will be giving more information of how I fix the issue, however let’s learn some basic terms and general information about ingress controller.

What is ingress controller ?

The best and simple explanation to this question is coming from Kubernetes official documentation over here, as they are expressing that ;

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

An [Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontend to help handle the traffic.

Whenever you have services which are running inside a cluster and would like to access them, you need to setup ingress controller for that cluster. The missing part was having no ingress controller on worker nodes in my k8s cluster. Everything was working however there was no access to them from outside world, that’s why ingress controller should take place in cluster architecture.

In this post, I will go for NGINX ingress controller with its default setup, however there are plenty of different ingress controllers which you may go for. I might change NGINX to Traefik in future but it depends on requirements yet for now, I will go with nginx ingress controller. The reason is that, it is super easy to setup, super rich with different features, included Kubernetes official documentation and fulfill what I am expecting for now.

Updates to cluster

Let’s briefly what I have explained in previous post;

  • Create VMs
  • Setup SSH connection
  • Use KubeSpray to deploy cluster
  • Create HAProxy and establish SSH connection with all nodes.

I have noticed that when deploying cluster, some add-ons should be enabled in order to use ingress controller from cluster with external HAProxy load balancer. Now, since cluster deployment was established with Ansible playbooks, it is not needed to setup everything from scratch. All modified configuration can be re-deployed without effecting any resource which is exists on cluster setup. It means that, I can enable required parts in configuration file and re-deploy cluster as I did on previous post.

  • Enable ingress controller from inventory file inside KubeSpray

Once this configuration part is updated from existing KubeSpray configuration files, k8s cluster should be redeployed with same command in previous post

*Assumption* : previous configured KubeSpray settings are used.

It will take a while and update all necessary parts which are required.

  • Include Ingress API object to route traffic from external HAProxy server to internal services

To include Ingress API object, HAProxy configuration file should be modified, following lines should be added to /etc/haproxy/haproxy.cfg file.

In given configuration balancing algorithm is `leastconn` which can be changed into any load balancer algorithm which is supported by HAProxy, however `leastconn` algorithm is fitting more to what I would like to achieve that’s why it is declared as `leastconn`. Note that this configuration addition is on top of added part on previous post.

Once HAProxy configuration is updated, HAProxy should be restarted `systemctl restart haproxy`. It is all for HAProxy configuration, now let’s dive into setting up NGINX Ingress Controller.

  • Setup NGINX Ingress Controller

It is super simple to deploy and setting up NGINX ingress controller since it is well documented and explains required parts in detail. To setup NGINX Ingress Controller, I will follow official guideline which is exists on NGINX Ingress Controller Installation.

NGINX Ingress Controller

Image is taken from https://www.nginx.com/products/nginx/kubernetes-ingress-controller/#resources

In normal cases, the situation is as given figure above, however, since in existing k8s cluster, I am using HAProxy for communicating with clients, I need NGINX ingress controller inside worker nodes which will manage running applications/services by communicating with HAProxy and eventually, the services will be accessible from outside world.

If I summarize how overview diagram will look like in my case is like in given figure below.

Overview of HA Kubernetes Cluster
Overview of HA Kubernetes Cluster
Overview of HA k8s cluster

It can be observed that, in given k8s cluster overview, HAProxy is in front, it communicates with clients, afterwards transmitting request based on defined rule on HAProxy configuration. Each worker node has NGINX ingress controller, what exactly it means, whenever a request appear to cluster, worker nodes will agree between each other and response back to user without having any problem. Since NGINX ingress controller is capable of load balancing inside worker nodes as well.

There is also Ingress Resource Rules part inside cluster, what it does, is that all routing rules based on path forwarded given service, an example on this is given below.

Steps to create NGINX Ingress controller

All steps shown below for installation of NGINX Ingress Controller taken from https://docs.nginx.com/nginx-ingress-controller/installation/

Make sure that you are a client with administrator privilege, all steps related to NGINX ingress controller should be done through `kubectl` (on client computer/server)

  • Clone Ingress Controller Repo
  • Create a namespace and a service account for the Ingress controller
  • Create a cluster role and cluster role binding for the service account
  • Create a secret with a TLS certificate and a key for the default server in NGINX

- **Create a config map for customizing NGINX configuration:**

Afterwards, there are two different ways to run NGINX ingress controller deployment, which are as daemonset or as deployment. Main difference between those are summarized on official installation page as;

Use a Deployment. When you run the Ingress Controller by using a Deployment, by default, Kubernetes will create one Ingress controller pod.

Use a DaemonSet: When you run the Ingress Controller by using a DaemonSet, Kubernetes will create an Ingress controller pod on every node of the cluster.

I will go with DaemonSet approach, the reason is that generally when you have background tasks which will run non-stateless then DaemonSet is more preferred way of running it.

Once it is applied as daemon set, the result could be checked with following command and result will be similar to given result below.

Deploy Example Application

To test how an application will be exposed to externally from k8s cluster, an example applicaton could be deployed as given below. Note that the following example is simplest example for this context, hence, keep in mind that it might require more configuration and detailed approach then described here when you would like to deploy more complex applications.

  • Create a sample NGINX Web Server (Using provided example)

*nginx-deploy-main.yml*

Example NGINX Deployment

Taken from https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/

In given yaml deployment file above, two replicas of NGINX:1.14.2 will be deployed to cluster and it has name of `nginx-deployment`. The yaml explains itself very well.

It can be deployed either through directly from official link or from your local depends on your preferences.

Expose deployment:

Once it is deployed to cluster and exposed, there is one step left for this simple counter example is that, exposing the service and creating ingress rule (resource) in yaml file, by specifiying kind as `Ingress`.

Ingress Ruleset

The crucial part is `serviceName` and `servicePort` which are defining specifications of the services within cluster. The yaml specifications can be expanded as shown below, assume that you have wildcard record in your domain name server and have multiple services which are running in same port in a cluster, yaml file can be re-defined as given below.

*nginx-ingress-resource.yml*

Ingress rules for different services

Keep in mind that all given services should be deployed before hand otherwise when a request made to any path which is not deployed, it may return either 404 or 500. There are plenty of different options to define and update the components in a k8s cluster. Therefore, all yaml files should be changed according to requirements.

*Create ingress controller rules from provided yaml file*

Now, the NGINX web server deployment is ready on given DNS record in yaml file and according to request paths different services can be called which are also running inside kubernetes cluster.

Example NGINX Web Server Deployment Result

Note that, provided yaml files are just simple example of deploying NGINX web server without any certification, when certificates (HTTPS) enabled or any other type of deployment happened different configurations should be applied.

When everything goes without any problem, you will have a cluster which uses NGINX Ingress controller for internal cluster routing and HAProxy as communication endpoint for clients. Keep in mind that whenever a new service or deployment take place, required configuration should be enabled in HAProxy configuration as it is enabled for port 80 applications above. Different services will have different requirements therefore it is important to catch main logic in a setup. It is all done for this post.

Cheers !

Software Engineer at Aalborg University, Denmark