Internal and external connectivity in Kubernetes space

By Rastko Vukasinovic

When getting into space where we are managing more than one web server with multiple different sets of pods, above mentioned services turn out to be quite complex to manage in most of the real life cases.

Let’s review example we had before — 2 APIs, redis and frontend, and imagine that APIs have more consumers then just frontend service so they need to be exposed to open internet.

Requirements are as following:

Setup needed using above services:

  • ClusterIP service to make components easily accessible to each other within the cluster.
  • NodePort service to expose some of the services outside of node, or maybe
  • LoadBalacer service if in the cloud, or
  • proxy server like nginx, to connect and route everything properly (30xxx ports to port 80, different services to paths on the proxy etc)
  • Deciding on where to do SSL implementation and maintaining it across

So

ClusterIP is necessary, we know it has to be there — it is the only one handling internal networking, so it is as simple as it can be.

External traffic however is different story, we have to set up at least one service per component plus one or multiple supplementary services (load balancers and proxies) in order to achieve requirements.

Number of configs / definitions to be maintained skyrockets, entropy rises, infrastructure setup drowns in complexity…

Solution

Kubernetes cluster has ingress as a solution to above complexity. Ingress is, essentially, layer 7 load balancer.

Layer 7 load balancer is name for type of load balancer that covers layers 5,6 and 7 of networking, which are session, presentation and application

Ingress can provide load balancing, SSL termination and name-based virtual hosting.

It covers HTTP, HTTPS.

For anything other then HTTP and HTTPS service will have to be published differently through special ingress setup or via a NodePort or LoadBalancer, but that is now a single place, one time configuration.

Ingress setup

In order to setup ingress we need two components:

  • Ingress controller — component that manages ingress based on provided rules
  • Ingress resources — Ingress HTTP rules

Ingress controller

There are few options you can choose from, among them nginx, GCE (google cloud) and Istio. Only two are officially supported by k8s for now — nginx and GCE.

We are going to go with nginx as the ingress controller solution. For this we, of course, need new deployment.

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginx-ingress-controllerspec: replicas: 1 selector: matchLabels: name: nginx-ingress template: metadata: labels: name: nginx-ingress spec: containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller args: - /nginx-ingress-controller - configMap=$(POD_NAMESPACE)/ingress-config env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https

containerPort: 443

Deploy ConfigMap in order to control ingress parameters easier:

apiVersion: v1kind: ConfigMapmetadata:

name: nginx-configuration

Now, with basic deployment in place and ConfigMap to make it easier for us to control parameters of the ingress, we need to setup the service to expose ingress to open internet (or some other smaller network).

For this we setup node port service with proxy/load balancer on top (bare-metal /on-prem example) or load balancer service (Cloud example).

In both mentioned cases, there is a need for Layer 4 and Layer 7 load balancer:

  • NodePort and possibly custom load balancer on top as L4 and Ingress as L7.
  • LoadBalancer as L4 and Ingress as L7.

Layer 4 load balancer — Directing traffic from network layer based on IP addresses or TCP ports, also referred to as transport layer load balancer.

NodePort for ingress yaml, to illustrate the above:

apiVersion: v1kind: Servicemetadata: name: nginx-ingressspec: type: NodePort ports: -targetPort: 80 port: 80 protocol: TCP name: http -targetPort: 433 port: 433 protocol: TCP name: httpsselector:

name: nginx-ingress

This NodePort service gets deployed to each node containing ingress deployment, and then load balancer distributes traffic between nodes

What separates ingress controller from regular proxy or load balancer is additional underlying functionality that monitors cluster for ingress resources and adjusts nginx accordingly. In order for ingress controller to be able to do this, service account with right permissions is needed.

apiVersion: v1kind: ServiceAccountmatadata:

name: nginx-ingress-serviceaccount

Above service account needs specific permissions on cluster and namespace in order for ingress to operate correctly, for particularities of permission setup on RBAC enabled cluster look at this document in nginx ingress official docs.

When we have all the permissions set up, we are ready to start working on our application ingress setup.

Ingress resources

Ingress resources configuration lets you fine tune incoming traffic (or fine-route).

Let’s first take simple API example. Assuming that we have just one set of pods deployed and exposed through service named simple-api-service on port 8080, we can create simple-api-ingress.yaml.

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: simple-api-ingressspec: backend: serviceName: simple-api-service

servicePort: 8080

When we kubectl create -f simple-api-ingress.yaml we setup an ingress that routes all incoming traffic to simple-api-service.

Rules

Rules are providing configuration to route incoming data based on certain conditions. For example, routing traffic to different services within the cluster based on subdomain or a path.

Let us now get to initial example:

  • frontend lives on www.example.com and everything not /api
  • api 1 is search api at www.example.com/api/search
  • api 2 is general (everything else) api that lives on www.example.com/api

Since everything is on the same domain, we can handle it all through one rule:

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: proper-api-ingressspec: rules: -http: paths: -path: /api/search backend: serviceName: search-api-service servicePort: 8081 -path: /api backend: serviceName: api-service servicePort: 8080 -path: / backend: serviceName: frontend-service

servicePort: 8082

There is also a default backend that is used to serve default pages (like 404s) and it can be deployed separately. In this case we will not need it since frontend will cover 404s.

You can read more at https://kubernetes.io/docs/concepts/services-networking/ingress/

Illustration of multi service kubernetes deployment being connected internally through ClusterIP and externally via ingress.
Illustration of multi service kubernetes deployment being connected internally through ClusterIP and externally via ingress.
Managing ingress and internal traffic in kubernetes

And, what if we changed the example to:

  • frontend lives on app.example.com
  • api 1 is search api at api.example.com/search
  • api 2 is general (everything else) api that lives on api.example.com

It is also possible, with the introduction of a new structure in the rule definition:

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: proper-api-ingressspec: rules: -host: api.example.com http: paths: -path: /search backend: serviceName: search-api-service servicePort: 8081 -path: / backend: serviceName: api-service servicePort: 8080 -host: app.example.com http: paths: -path: / backend: serviceName: frontend-service

servicePort: 8082

Note (out of scope): You can notice from the last illustration that there are multiple ingress pods, which implies that ingress can scale, and it can. Ingress can be scaled like any other deployment, you can also have it auto scale based on internal or external metrics (external, like number of requests handled is probably the best choice).

Note 2 (out of scope): Ingress can, in some cases, be deployed as DaemonSet, to assure scale and distribution across the nodes.

Wrap

This was a first pass through the structure and usage of k8s services and networking capabilities that we need in order to structure communication inside and outside of the cluster.

I, as always, tried to provide to the point and battle tested guide to reality… What is written above should give you enough knowledge to deploy ingress and setup basic set of rules to route traffic to your app and give you context for further fine tuning of your setup.

Important piece of advice: Make sure to keep all the setups as a code in files, in your repo — infrastructure as a code is essential part of making your application reliable.