Skip to main content

Develop Applications to integrate into ClusterFactory

This guide covers the general process to develop an application for Cluster Factory and might help you to integrate your applications.

Let's take the example of xCAT, which is a complex bare-metal provisioning solution.

1. Dockerize/Containerize the application

All applications are containerizable. You will need to fetch a lot of information to see how difficult it is.

You should check for:

  • Dependencies:
    • The base image (Ex: Rocky Linux)
    • Build-time setup (enabling the services)
    • Runtime dependencies (Ex: Perl, Apache, xCAT, ...)
    • Runtime setup (the entry-point script)
    • Init system (SystemD)
    • And eventually, host dependencies
  • Interfaces:
    • Network:
      • All the TCP and UDP ports (Ex: DHCP, SSH, ...)
      • Host network (Ex: The DHCP server needs the host network to receive broadcast DHCP requests.)
    • Volumes:
      • Persistent Volumes (Ex: The xCAT databases.)
      • Is it possible to set a read-only filesystem?
  • Privileges
    • Is it possible to run rootless?
    • Is there any capabilities? (Ex: NET_BIND_SERVICE, ...)

Knowing these details will make it easier to write a Dockerfile and test it.

The xCAT Dockerfile:

FROM quay.io/rockylinux/rockylinux:8.4

LABEL MAINTAINER Square Factory

ENV container docker

ARG xcat_version=latest
ARG xcat_reporoot=https://xcat.org/files/xcat/repos/yum
ARG xcat_baseos=rh8

# Remove useless SystemD services
RUN (cd /lib/systemd/system/sysinit.target.wants/; \
for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/* \
&& rm -f /etc/systemd/system/*.wants/* \
&& rm -f /lib/systemd/system/local-fs.target.wants/* \
&& rm -f /lib/systemd/system/sockets.target.wants/*udev* \
&& rm -f /lib/systemd/system/sockets.target.wants/*initctl* \
&& rm -f /lib/systemd/system/basic.target.wants:/* \
&& rm -f /lib/systemd/system/anaconda.target.wants/*

# Setup symlink
RUN mkdir -p /xcatdata/etc/{dhcp,goconserver,xcat} && ln -sf -t /etc /xcatdata/etc/{dhcp,goconserver,xcat} && \
mkdir -p /xcatdata/{install,tftpboot} && ln -sf -t / /xcatdata/{install,tftpboot}

# Install dependencies
RUN dnf install -y -q wget which \
&& wget ${xcat_reporoot}/${xcat_version}/$([[ "devel" = "${xcat_version}" ]] && echo 'core-snap' || echo 'xcat-core')/xcat-core.repo -O /etc/yum.repos.d/xcat-core.repo \
&& wget ${xcat_reporoot}/${xcat_version}/xcat-dep/${xcat_baseos}/$(uname -m)/xcat-dep.repo -O /etc/yum.repos.d/xcat-dep.repo \
&& dnf install -y \
xCAT \
openssh-server \
rsyslog \
createrepo \
chrony \
initscripts \
man \
nano \
pigz \
bash-completion \
vim \
epel-release \
&& dnf install -y \
screen \
bind-utils \
&& dnf clean all

# Setup SSH
RUN sed -i -e 's|#PermitRootLogin yes|PermitRootLogin yes|g' \
-e 's|#Port 22|Port 2200|g' \
-e 's|#UseDNS yes|UseDNS no|g' /etc/ssh/sshd_config \
&& echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config \
&& echo "root:cluster" | chpasswd \
&& rm -rf /root/.ssh \
&& mv /xcatdata /xcatdata.NEEDINIT

# Enable services
RUN systemctl enable httpd \
&& systemctl enable sshd \
&& systemctl enable dhcpd \
&& systemctl enable rsyslog \
&& systemctl enable xcatd

# Copy our edited genimage
COPY ./opt/xcat/share/xcat/netboot/rh/genimage /opt/xcat/share/xcat/netboot/rh/genimage

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

COPY startup.sh /startup.sh
RUN chmod +x /startup.sh

ENV XCATROOT /opt/xcat
ENV PATH="$XCATROOT/bin:$XCATROOT/sbin:$XCATROOT/share/xcat/tools:$PATH" MANPATH="$XCATROOT/share/man:$MANPATH"
VOLUME [ "/xcatdata", "/var/log/xcat" ]

EXPOSE 3001/tcp 3001/udp \
3002/tcp 3002/udp \
7/udp \
873/tcp 873/udp \
53/tcp 53/udp \
67/tcp 67/udp \
68/tcp 68/udp \
69/tcp 69/udp \
111/udp \
514/tcp 514/udp \
4011/tcp \
623/tcp 623/udp \
2200/udp

CMD [ "/startup.sh" ]

The EXPOSE declares which ports must be open for xCAT to be fully functional.

The VOLUME declares which volumes need to be persistent.

Other volumes can be mounted as read-only configurations. For example, since we are running systemd, we need to mount the /sys/fs/cgroup directory.

The entry point:

statup.sh
#!/bin/bash

setsid ./entrypoint.sh &

exec /sbin/init
entrypoint.sh
#!/bin/bash
is_ubuntu=$(test -f /etc/debian_version && echo Y)
[[ -z ${is_ubuntu} ]] && logadm="root:" || logadm="syslog:adm"
chown -R ${logadm} /var/log/xcat/
. /etc/profile.d/xcat.sh
ps -ax
if [[ -d "/xcatdata.NEEDINIT" ]]; then
echo "initializing xCAT ..."
if [ ! -f "/xcatdata/.init-finished" ]; then
echo "first initalization, copying template..."
rsync -a /xcatdata.NEEDINIT/ /xcatdata

echo "initalizing database."
xcatconfig --database

touch /xcatdata/.init-finished
fi

echo "initializing networks table if necessary..."
xcatconfig --updateinstall
XCATBYPASS=1 tabdump site | grep domain || XCATBYPASS=1 chtab key=domain site.value=example.com

if ! [ -L /root/.xcat ]; then
if ! [ -d /xcatdata/.xcat ]; then
echo "backup data not found, regenerating certificates and copying..."
xcatconfig -c
rsync -a /root/.xcat/* /xcatdata/.xcat
fi
echo "create symbol link for /root/.xcat..."
rm -rf /root/.xcat/
ln -sf -t /root /xcatdata/.xcat
fi

if [ -d /xcatdata/.ssh ]; then
echo "copy backup keys in /root/.ssh..."
rsync -a /xcatdata/.ssh/ /root/.ssh/
chmod 600 /root/.ssh/*
else
echo "backup keys not found, copying keys to /xcatdata/.ssh..."
xcatconfig --sshkeys
mkdir -p /xcatdata/.ssh
rsync -a /root/.ssh/ /xcatdata/.ssh/
chmod 600 /xcatdata/.ssh/*
fi

echo "reconfiguring hosts..."
makehosts
echo "reconfiguring dns..."
makedns
echo "reconfiguring dhcpd config..."
makedhcp -n
echo "reconfiguring dhcpd leases..."
makedhcp -a

echo "initializing loop devices..."
# workaround for no loop device could be used by copycds
for i in {0..7}; do
test -b /dev/loop$i || mknod /dev/loop$i -m0660 b 7 $i
done
# workaround for missing `switch_macmap` (#13)
ln -sf /opt/xcat/bin/xcatclient /opt/xcat/probe/subcmds/bin/switchprobe
mv /xcatdata.NEEDINIT /xcatdata.orig
fi

cat /etc/motd
HOSTIPS=$(ip -o -4 addr show up | grep -v "\<lo\>" | xargs -I{} expr {} : ".*inet \([0-9.]*\).*")
echo "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"
echo "welcome to Dockerized xCAT, please login with"
[[ -n "$HOSTIPS" ]] && for i in $HOSTIPS; do echo " ssh root@$i -p 2200 "; done && echo "The initial password is \"cluster\""
echo "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"

systemctl start xcatd
#exec /sbin/init
rm -f /etc/nologin /var/run/nologin

2. Testing the application with Podman

Podman is an alternative to Docker. The main difference is that Podman is daemon-less.

We will focus on one specific feature which is podman-play-kube.

While you might test the container with Docker with docker-compose or with Minikube, Podman offers the almost same experience as a Kubernetes Cluster without being overkill.

podman-play-kube only supports Pod, Deployment, PersistentVolumeClaim and ConfigMap, but that's enough since it bridges the gap between docker-compose and the Kubernetes syntax.

Let's write a Pod for xCAT:

pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: 'xcat'
namespace: default
labels:
app: 'xcat'
spec:
hostNetwork: true
containers:
- name: xcat
image: 'xcat:latest'
imagePullPolicy: Never
securityContext:
capabilities:
add:
- CAP_SYS_ADMIN
- NET_ADMIN
readOnlyRootFilesystem: false
runAsNonRoot: false
runAsUser: 0
resources:
limits:
cpu: 200m
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
ports:
- name: xcatdport-tcp
containerPort: 3001
protocol: TCP
- name: xcatdport-udp
containerPort: 3001
protocol: UDP
- name: xcatiport-tcp
containerPort: 3002
protocol: TCP
- name: xcatiport-udp
containerPort: 3002
protocol: UDP
- name: echo-udp
containerPort: 7
protocol: UDP
- name: rsync-tcp
containerPort: 873
protocol: TCP
- name: rsync-udp
containerPort: 873
protocol: UDP
- name: domain-tcp
containerPort: 53
protocol: TCP
- name: domain-udp
containerPort: 53
protocol: UDP
- name: bootps
containerPort: 67
protocol: UDP
- name: dhcp
containerPort: 67
protocol: TCP
- name: dhcpc
containerPort: 68
protocol: TCP
- name: bootpc
containerPort: 68
protocol: UDP
- name: tftp-tcp
containerPort: 69
protocol: TCP
- name: tftp-udp
containerPort: 69
protocol: UDP
- name: www-tcp
containerPort: 80
protocol: TCP
- name: www-udp
containerPort: 80
protocol: UDP
- name: sunrpc-udp
containerPort: 111
protocol: UDP
- name: rsyslogd-tcp
containerPort: 514
protocol: TCP
- name: rsyslogd-udp
containerPort: 514
protocol: UDP
- name: pxe
containerPort: 4011
protocol: TCP
- name: ipmi-tcp
containerPort: 623
protocol: TCP
- name: ipmi-udp
containerPort: 623
protocol: UDP
- name: ssh-tcp
containerPort: 2200
protocol: TCP
- name: ssh-udp
containerPort: 2200
protocol: UDP
volumeMounts:
- name: xcatdata
mountPath: /xcatdata
- name: cgroup
mountPath: /sys/fs/cgroup
readOnly: true
- name: varlogxcat
mountPath: /var/log/xcat
- mountPath: /tmp
name: tmp
subPath: tmp
- mountPath: /run
name: tmp
subPath: run
- mountPath: /run/lock
name: tmp
subPath: run-lock
volumes:
- name: tmp
hostPath:
path: ./tmp
- name: varlogxcat
hostPath:
path: ./logs
- name: xcatdata
hostPath:
path: ./xcat
- name: cgroup
hostPath:
path: /sys/fs/cgroup
type: Directory
restartPolicy: Always

If we were to write a docker-compose.yml, we would open the same port, mount the same volumes and add the same capabilities.

The main advantage is that if it works with podman, it will certainly work with Kubernetes.

One main disadvantage is that podman-play-kube doesn't support the use of networks (which means no support for macvlan and ipvlan). The issue is tracked here.

3. Writing a Helm application

Although it is not necessary to write a Helm application, some values may be redundant or must be abstracted.

That's why we prefer to write Helm Charts instead of Kustomize. If the application is light enough, we can use Kustomize instead.

To write a Helm application, we need to generalize the values (by using example.com as domain for example). The "overlay" values will be stored either inside a Git repository, more precisely, inside a fork.

The Helm application must be available on a publicly accessible Git or Helm repository.

The example for xCAT is stored inside helm/xcat.

4. Writing the Argo CD Application and custom values

After writing the Helm Chart, you can write the Argo CD Application.

The example for xCAT is stored inside argo/provisioning/apps.

The custom values are stored inside the helm/xcat directory. If the Helm application is not a Git repository, it's better to use the subchart pattern by using helm dependencies.

5. Testing on the Kubernetes cluster

Our favorite software for debugging is Lens.

If you have deployed Prometheus, we can see the CPU and Memory usage of the container.

image-20220517160622884