Kubernetes Lab on Baremetal

By Marco Lancini

In “Deploy Your Own Kubernetes Lab” I covered multiple deployment options for a Kubernetes lab, ranging from more lightweight (like running Kubernetes locally) to more realistic ones (like deploying a multi-node cluster) suitable for security research.

In this blog post, I’m going to detail the steps I took to deploy my own Kubernetes Lab on baremetal, and on an Intel NUC in particular.

The Hardware

I was looking for a self-contained option, which - most importantly - didn’t take up much space, so I ended up settling on an Intel NUC, starting with 250GB of storage and 32GB of RAM.

It might be worth noting that, for the initial setup phase, it is also useful to have a small keyboard (like this one) and a monitor (a 7inch one is just fine) around.

At a high level, my home network diagram looks like the one below:

High Level Network Setup.
High Level Network Setup.

Install CoreOS

As the title of this post implies, the aim was to have a Kubernetes cluster running directly on baremetal, hence deciding which operating system to rely on was almost straightforward: Fedora CoreOS (FCOS) is a minimal operating system specifically designed for running containerized workloads securely and at scale.

Let’s see how to get it running on the Intel NUC.

Prepare a Bootable USB

First step in the installation process involves burning a Fedora CoreOS ISO onto a bootable USB stick.

The latest stable version of the ISO for baremetal installations can be found directly on the Fedora website (33.20210301.3.1 at the time of writing). From there, it is simply a matter of burning the ISO, which, on macOS, can be done using tools like Etcher. Once launched, select the CoreOS ISO and the USB device to use, and Etcher will take care of creating a bootable USB from it.

Using Etcher to Create a Bootable USB Stick.
Using Etcher to Create a Bootable USB Stick.

Prepare an Ignition Config

For those new to FCOS (me included before creating this lab), it might be worth explaining what an Ignition file actually is. An Ignition file specifies the configuration for provisioning FCOS instances: the process begins with a YAML configuration file, which gets translated by the FCOS Configuration Transpiler (fcct) into a machine-friendly JSON, which is the final configuration file for Ignition. FCOS ingests the Ignition file only on first boot, applying the whole configuration or failing to boot in case of errors.

The Fedora documentation proved to be excellent in detailing how to create a basic Ignition file that modifies the default FCOS user (named core) to allow logins with an SSH key.

First, on your workstation create a file (named config.fcc) with the following content, and make sure to replace the line starting with ssh-rsa with the contents of your SSH public key file:

➜ cat config.fcc
variant: fcos
version: 1.3.0
passwd: users: - name: core groups: - docker - wheel - sudo ssh_authorized_keys: - ssh-rsa AAAA...

In the config above, we are basically telling FCOS to add the default user named core to three additional groups (docker, wheel, and sudo), as well as to allow key based authentication with the the public SSH key specified in the ssh_authorized_keys section. The public key will be provisioned to FCOS machine via Ignition, whereas the private counterpart needs to be available to your user on your local workstation, in order to remotely authenticate over SSH.

Next, we need to use fcct, the Fedora CoreOS Config Transpiler, to produces a JSON Ignition file from a YAML FCC file. An easy way to use fcct is to run it in a container:

➜ docker run --rm -i quay.io/coreos/fcct:release --pretty --strict < config.fcc > config.ign
➜ cat config.ign
{ "ignition": { "version": "3.2.0" }, "passwd": { "users": [ { "groups": [ "docker", "wheel", "sudo" ], "name": "core", "sshAuthorizedKeys": [ "ssh-rsa AAAA..." ] } ] }
}

Since this config.ign will be needed to boot FCOS, we need to make it temporarily available for devices on the local network. There are multiple ways to accomplish this: I did opt to quickly spin up updog (a replacement for Python’s SimpleHTTPServer):

Sharing the Ignition File over LAN.
Sharing the Ignition File over LAN.

Install from Live USB

With the Ignition config ready, plug the USB stick in the Intel NUC, turn it on, and make sure to select that media as preferred boot option. If the ISO has been burnt correctly, you should end up in a shell as the core user.

The actual installation can be accomplished in a quite straightforward way with coreos-installer:

$ sudo coreos-installer install /dev/sda \ --insecure-ignition --ignition-url http://192.168.1.150/config.ign

The command above instructs coreos-installer to use the Ignition config we are making available to local network from our workstation (192.168.1.150 in my case). The --insecure-ignition flag is needed if the Ignition file is served over plaintext HTTP rather than TLS.

After a reboot of the Intel NUC, you should be able to SSH into it from your workstation:

❯ ssh [email protected]
Fedora CoreOS 33.20210217.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/ [[email protected] ~]$ id
uid=1000(core) gid=1000(core) groups=1000(core),4(adm),10(wheel),16(sudo),190(systemd-journal)

And that’s it! FCOS is now up and running. Next step is installing Kubernetes on it.

Install Kubernetes

The installation process for Kubernetes is a bit more lenghty, and can be broken up in a few sections: installation of dependencies, installation of the cluster, and network setup.

Install Dependencies

While looking around (i.e., Googling) for the most effective way to deploy a vanilla Kubernets on FCOS I came across a really detailed article from Matthias Preu (Fedora CoreOS - Basic Kubernetes Setup) describing exactly this process. Note that the remainder of this sub-section has been based heavily on Matthias’ setup, and you should refer to his blog post for a detailed explanation of each installation step.

First, setup CRI-O as the container runtime:

# Activating Fedora module repositories
$ sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-modular.repo
$ sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-updates-modular.repo
$ sed -i -z s/enabled=0/enabled=1/ /etc/yum.repos.d/fedora-updates-testing-modular.repo # Setting up the CRI-O module
$ mkdir /etc/dnf/modules.d
$ cat <<EOF > /etc/dnf/modules.d/cri-o.module
[cri-o]
name=cri-o
stream=1.17
profiles=
state=enabled
EOF # Installing CRI-O
$ rpm-ostree install cri-o
$ systemctl reboot $ modprobe overlay && modprobe br_netfilter
$ cat <<EOF > /etc/modules-load.d/crio-net.conf
overlay
br_netfilter
EOF $ cat <<EOF > /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF $ sysctl --system
$ sed -i -z s+/usr/share/containers/oci/hooks.d+/etc/containers/oci/hooks.d+ /etc/crio/crio.conf

Next, install all tooling required to manage the cluster (kubeadm, kubelet and kubectl):

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF $ rpm-ostree install kubelet kubeadm kubectl
$ systemctl reboot $ setenforce 0
$ sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config $ systemctl enable --now cri-o && systemctl enable --now kubelet
$ echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet

Install the Cluster

Before starting the installation of the cluster itself, a custom cluster configuration needs to be created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[[email protected] core]$ cat <<EOF > clusterconfig.yml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.5
controllerManager: extraArgs: flex-volume-plugin-dir: "/etc/kubernetes/kubelet-plugins/volume/exec"
networking: podSubnet: 10.244.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration: criSocket: /var/run/crio/crio.sock
EOF
  • Line 4: the Kubernetes version to deploy (1.20.5 in my case).
  • Line 9: the subnet to be used to allocate pods’ IP addresses. Pay attention that the prefix 10.244.X.X/X is required by Flannel (the chosen networking solution) when used in conjunction with kubeadm.

With the config ready, we can use kubeadm to install the cluster:

[[email protected] core]$ kubeadm init --config clusterconfig.yml [init] Using Kubernetes version: v1.20.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cluster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.151]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [cluster localhost] and IPs [192.168.1.151 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [cluster localhost] and IPs [192.168.1.151 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 70.502118 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node cluster as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node cluster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 9fcige.wjsr2lub81pr86tc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.1.151:6443 --token <redacted> \ --discovery-token-ca-cert-hash sha256:<redacted>

As it can be seen from the output of kubeadm itself, we can now grant the core user (or any local user, actually) access to the cluster by copying the kubeconfig file to its .kube directory:

[[email protected] ~]$ mkdir -p $HOME/.kube
[[email protected] ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[[email protected] ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

The same config can also be copied (e.g., via scp) onto your workstation, so to interact with the cluster remotely without having to SSH into the NUC:

➜ kubectx cluster # alias for the cluster
➜ k cluster-info
+ kubectl cluster-info
Kubernetes control plane is running at https://192.168.1.151:6443
KubeDNS is running at https://192.168.1.151:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy ➜ kg nodes -o wide
+ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cluster Ready control-plane,master 60m v1.20.5 192.168.1.151 <none> Fedora CoreOS 33.20210301.3.1 5.10.19-200.fc33.x86_64 cri-o://1.19.1

From the output above you can see how the control plane is reachable at the NUC’s local IP address (192.168.1.151 in my case).

Network Setup

Although it might seem everything is setup, there are still a couple of steps missing.

First, since I only have one node available, it is necessary to allow the master node itself to schedule pods. This is done by removing a taint:

➜ k taint nodes --all node-role.kubernetes.io/master-

Second, we need to deploy a networking solution like Flannel:

[[email protected] ~]$ sudo sysctl net.bridge.bridge-nf-call-iptables=1
[[email protected] ~]$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

After this, you should have all the necessary components for a basic Kubernetes cluster up and running:

➜ kgpo --all-namespaces
+ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-74ff55c5b-2qdkf 1/1 Running 0 3d17h
kube-system coredns-74ff55c5b-5blfn 1/1 Running 0 3d17h
kube-system etcd-cluster 1/1 Running 0 3d17h
kube-system kube-apiserver-cluster 1/1 Running 0 3d17h
kube-system kube-controller-manager-cluster 1/1 Running 0 3d17h
kube-system kube-flannel-ds-22ltx 1/1 Running 0 3d17h
kube-system kube-proxy-2lbvn 1/1 Running 0 3d17h
kube-system kube-scheduler-cluster 1/1 Running 0 3d17h