GitOps with Grendel

Postscript strategy

If you've looked inside the Packer recipes, you can see that there is a systemd service that will run to fetch the postscript on Grendel:

Extract of ks.bare.cfg

cat << 'END' >/pull-postscript.sh
#!/bin/sh
set -ex

HOSTNAME="$(sed -E 's/^.*grendel.hostname=([^ ]*).*$/\1/' /proc/cmdline)"
hostnamectl set-hostname "${HOSTNAME}"

GRENDEL_ADDRESS="$(sed -E 's/^.*grendel.address=([^ ]*).*$/\1/' /proc/cmdline)"

curl -fsSL ${GRENDEL_ADDRESS}/repo/postscript.sh -o /postscript.sh
chmod +x /postscript.sh
/postscript.sh ${HOSTNAME}
END

chmod +x /pull-postscript.sh

cat <<'END' >/etc/systemd/system/grendel-postscript.service
[Unit]
Description=Grendel Postscript
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/pull-postscript.sh

[Install]
WantedBy=multi-user.target
END
ln -s "/etc/systemd/system/grendel-postscript.service" "/etc/systemd/system/multi-user.target.wants/grendel-postscript.service"

The postscript is defined inside the Grendel configuration:

helm/grendel/values-production.yaml

config:
  postscript: ''

The strategy to enable GitOps is the following:

1. The systemd service pull the grendel postscript.
2. The grendel postscript fetches the ssh private key from the grendel HTTP server.
3. The grendel postscript git clone a repository containing other postscripts by using the ssh private key.
4. After cloning the repository, grendel executes the postscripts.

GitHub configuration

You first postscript tracked with Git

Create a private Git repository for your scripts and add a post.sh script.

This script is the main entry point. If you want to add a hierarchy, you use this script:

Example of postscript

#!/bin/sh

# Find all the executable scripts and sort them by name
scripts=$(find ./scripts -type f | sort)

# Loop through each script and execute it
for script in $scripts; do
  # Check if the script needs to be chmod-ed
  if [ ! -x "$script" ]; then
    chmod +x "$script"
  fi

  # Execute the script
  "./$script"
done

This script will execute all the files inside the scripts folder in alphabetical order. So you need to create a scripts folder with scripts inside.

Commit and push everything.

Adding a deploy key

Generate a key pair using:

ssh-keygen -f $(pwd)/id_rsa -C grendel

And add the id_rsa.pub as a deploy key.

Node Configuration: Sending the private key to the compute nodes

Assuming /dev/nvme0n1 is the disk storing the private key, add the private key to the disk of the compute nodes:

root@cn1:/root

mkdir -p /secret
mount /dev/nvme0n1 /secret
cat << 'EOF' > /secret/key
-----BEGIN OPENSSH PRIVATE KEY-----
<your private key>
-----END OPENSSH PRIVATE KEY-----
EOF
chmod 600 /secret/key
umount /secret

This key will be protected via Linux permissions and will be accessible only by the root user.

This step has to be done manually.

Grendel configuration

Setup the Grendel postscript for GitOps

In the Grendel values file, change the postscript field to:

helm/grendel/values-production.yaml

config:
  postscript: |
    #!/bin/sh

    set -ex

    # Fetch deploy key
    mkdir -p /secret
    mount /dev/nvme0n1 /secret  # HERE!!!!: Change the device with the disk which will store the private key.

    # Cloning git repo containing postscripts.
    mkdir -p /configs
    GIT_SSH_COMMAND='ssh -i /secret/key -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' git clone git@github.com:<repo owner>/<repo>.git /configs
    if [ -f /configs/post.sh ] && [ -x /configs/post.sh ]; then
      cd /configs || exit 1
      ./post.sh "$1"
    fi

    # Security
    chmod -R g-rwx,o-rwx .
    umount -f /secret

Conclusion

And that's it! With this, the node postscripts will be tracked on Git and you won't be lost in your node configuration.