Skip to main content

3. K0s Configuration

Specifying the hosts

Specify the nodes that will be included in the Kubernetes Cluster, which will function as the control plane for managing compute nodes.

A node designated as a controller will run the following components:

  • An etcd server, which serves as the Kubernetes database.
  • An API server, which serves as the entry point for kubectl commands.
  • A Pod scheduler.
  • A controller-manager, which acts as the central decision-making component of Kubernetes.
  • A Konnectivity-server, responsible for facilitating communication between Kubernetes controller and worker nodes.
  • A k0s API, which serves as the entry point for k0s commands.

A node designated as a worker will solely run the following components:

  • A kubelet, which acts as an agent to communicate with the Konnectivity server
  • Containerd containers, which refer to Pods running on the worker node.

It is crucial to always have an odd number of controllers (1, 3, 5, ...) to prevent the cluster from getting stuck in a deadlock.

Edit the cfctl.yaml file. Start with the hosts field:

cfctl.yaml
apiVersion: cfctl.clusterfactory.io/v1beta1
kind: Cluster
metadata:
name: k8s.example.com-cluster
spec:
hosts:
- ssh:
address: 192.168.0.2
user: root
port: 22
keyPath: ~/.ssh/id_ed25519
role: controller+worker
noTaints: true
privateInterface: eno1
privateAddress: 192.168.0.2
installFlags:
- --debug
- --labels="topology.kubernetes.io/region=ch-sion,topology.kubernetes.io/zone=ch-sion-1"
- --disable-components coredns
hooks:
apply:
before:
# Set SELinux Permissive
- sh -c 'if [ "$(getenforce)" != "Permissive" ] && [ "$(getenforce)" != "Disabled" ]; then sed -i s/^SELINUX=.*$/SELINUX=permissive/ /etc/selinux/config; fi'
- sh -c 'if [ "$(getenforce)" != "Permissive" ] && [ "$(getenforce)" != "Disabled" ]; then setenforce 0; fi'

...

Provide each host with a valid IP address that is reachable by k0ctl, and the connection details for an SSH connection. Edit the labels for multi-zone usage.

See cfctl.yaml specification.

Configuring the k0s architecture

After you set the hosts field, you must configure the k0s architecture by editing the k0s field:

cfctl.yaml > spec > k0s
k0s:
version: '1.28.3+k0s.0'
dynamicConfig: false
config:
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
name: k8s.example.com
spec:
api:
k0sApiPort: 9443
port: 6443
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
konnectivity:
adminPort: 8133
agentPort: 8132
network:
calico:
mode: 'vxlan'
overlay: Always
mtu: 0
wireguard: false
kubeProxy:
disabled: false
mode: iptables
kuberouter:
autoMTU: true
mtu: 0
peerRouterASNs: ''
peerRouterIPs: ''
podCIDR: 10.244.0.0/16
provider: calico
serviceCIDR: 10.96.0.0/12
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
storage:
type: etcd
telemetry:
enabled: false

Most of the values are already sane, but you should check if the CIDR doesn't conflict with any IP range of your network. It is also recommended to tune manually the MTU and match it to your switch and router values.

If you wish to use a HA setup, please follow this guide.

Initial Deployment

tip

If you forgot to install the utilities, just run:

user@local:/ClusterFactory
. ./scripts/setup-env

Deploy the cluster with:

user@local:/ClusterFactory
# Deploy the cluster
cfctl apply --debug --config ./cfctl.yaml

# Fetch the kubeconfig
cfctl kubeconfig --config ./cfctl.yaml >./kubeconfig
chmod 600 ./kubeconfig

You can store the kubeconfig inside ~/.kube/config. Our recommendation is to set the KUBECONFIG environment variable to avoid mixing the Kubernetes contexts. Just like this:

user@local:/ClusterFactory
cfctl kubeconfig --config ./cfctl.yaml >./kubeconfig
chmod 600 ./kubeconfig
export KUBECONFIG=$(pwd)/kubeconfig

Just make sure to verify which configuration you are using with kubectl config current-context. You can add an alias to your favorite shell:

alias kubectx="kubectl config current-context"

Congratulation, you have deployed your Kubernetes cluster! However, it's still missing a few core features:

  • MetalLB advertisements, for Load Balancing
  • CoreDNS, which is the internal DNS for Kubernetes
  • KubeVirt, to deploy VM workloads
  • Multus CNI, to support multiple network interfaces
  • Sealed Secrets, secret management optimized for GitOps
  • Cert-manager issuers, to generate your SSL certificates and enable, for free, TLS configuration.
  • Argo CD, to enable GitOps.
  • Traefik, as an Ingress Controller