GitOps with Grendel

Postscript strategy

If you've looked inside the Packer recipes, you can see that there is a systemd service that will run to fetch the postscript on Grendel:

Extract of ks.bare.cfg

cat << 'END' >/pull-postscript.sh
#!/bin/sh
set -ex

HOSTNAME="$(sed -E 's/^.*grendel.hostname=([^ ]*).*$/\1/' /proc/cmdline)"
hostnamectl set-hostname "${HOSTNAME}"

GRENDEL_ADDRESS="$(sed -E 's/^.*grendel.address=([^ ]*).*$/\1/' /proc/cmdline)"

curl -fsSL ${GRENDEL_ADDRESS}/repo/postscript.sh -o /postscript.sh
chmod +x /postscript.sh
/postscript.sh ${HOSTNAME}
END

chmod +x /pull-postscript.sh

cat <<'END' >/etc/systemd/system/grendel-postscript.service
[Unit]
Description=Grendel Postscript
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/pull-postscript.sh

[Install]
WantedBy=multi-user.target
END
ln -s "/etc/systemd/system/grendel-postscript.service" "/etc/systemd/system/multi-user.target.wants/grendel-postscript.service"

The postscript is defined inside the Grendel configuration:

helm/grendel/values-production.yaml

config:
  postscript: ''

The strategy to enable GitOps is the following:

1. The systemd service pull the grendel postscript.
2. The grendel postscript fetches the ssh private key from the grendel HTTP server.
3. The grendel postscript git clone a repository containing other postscripts by using the ssh private key.
4. After cloning the repository, grendel executes the postscripts.

GitHub configuration

You first postscript tracked with Git

Create a private empty repository for your scripts and add a post.sh script.

This script is the main entry point. If you want to add a hierarchy, you use this script:

Example of postscript

#!/bin/sh

# Find all the executable scripts and sort them by name
scripts=$(find ./scripts -type f | sort)

# Loop through each script and execute it
for script in $scripts; do
  # Check if the script needs to be chmod-ed
  if [ ! -x "$script" ]; then
    chmod +x "$script"
  fi

  # Execute the script
  "./$script"
done

This script will execute all the files inside the scripts folder in alphabetical order. So you need to create a scripts folder with scripts inside.

Commit and push everything.

Adding a deploy key

Generate a key pair using:

ssh-keygen -f $(pwd)/id_rsa -C grendel

And add the id_rsa.pub as a deploy key.

Grendel configuration

Let's add the private key to the Grendel HTTP server.

Private key secret

Create a secret:

argo/provisioning/secrets/postscript-privatekey-secret.yaml.local

apiVersion: v1
kind: Secret
metadata:
  name: postscript-privatekey-secret
  namespace: provisioning
type: Opaque
stringData:
  ## Create the key with:
  ## ssh-keygen -f $(pwd)/key -C grendel
  key: ''

Seal it and apply it:

user@local:/ClusterFactory

cfctl kubeseal
kubectl apply -f argo/provisioning/secrets/postscript-privatekey-sealed-secret.yaml

Mounting the private key

In the Grendel values file, add:

helm/grendel/values-production.yaml

## Extra volumes
volumes:
  - name: postscript-privatekey
    secret:
      defaultMode: 384
      secretName: postscript-privatekey-secret

## Extra volume mounts
volumeMounts:
  - name: postscript-privatekey
    subPath: key
    mountPath: /var/lib/grendel/key

This will mount the key inside the HTTP server.

Setup the Grendel postscript for GitOps

In the Grendel values file, change the postscript field to:

helm/grendel/values-production.yaml

config:
  postscript: |
    #!/bin/sh

    set -ex

    # Fetch deploy key
    curl --retry 5 -fsSL http://grendel.internal/repo/key -o /key
    chmod 600 /key

    # Cloning git repo containing postscripts.
    mkdir -p /configs
    GIT_SSH_COMMAND='ssh -i /key -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' git clone git@github.com:<repo owner>/<repo>.git /configs
    if [ -f /configs/post.sh ] && [ -x /configs/post.sh ]; then
      cd /configs || exit 1
      ./post.sh "$1"
    fi
    rm -f /key

    # Security
    chmod -R g-rwx,o-rwx .

Conclusion

And that's it! With this, the node postscripts will be tracked on Git and you won't be lost in your node configuration.