MB

▄▄▄·▪ ▐ ▄ ▄▄▄ . ▐█ ▄███ •█▌▐█▀▄.▀· ██▀·▐█·▐█▐▐▌▐▀▀▪▄ ▐█▪·•▐█▌██▐█▌▐█▄▄▌ .▀ ▀▀▀▀▀ █▪ ▀▀▀

I am still working on a hard drive mount that won’t melt, but in the meantime the software side is ready. To nobody’s surprise, even this backup server is running Talos Linux managed by Omni. It is a two-node cluster. One worker node is the ZimaBlade that has the hard drive attached, and the one control plane node is a virtual one that is running on my other Omni cluster. You can run Talos as just a normal container, but not when managed by Omni. So it is actually a full QEMU virtual machine, but still running as a pod. Working with this virtual control plane means that the worker node is 100% stateless (except for the backup drive, of course). This helps make my offsite backup extra robust, because I will not always have physical access to the worker.

Because the worker and control plane are not on the same network, everything will need to be networked through NetBird. Luckily, this is easily done with the System Extensions of Talos. So NetBird is running as an OS process, which allows Flannel and Kubelet to talk over the virtual network. So the nodes can be anywhere, and things will just work, even behind symmetric NAT where KubeSpan won’t work. In NetBird, the Bunker cluster (that is what I am calling this small backup cluster) is fully isolated and has only connections between the nodes and a one-way connection from the worker node to my NAS (pull-based backups).

Next, what is running on the cluster. To start, my usual monitoring stack, but no storage of the metrics this time. I am running the Victoria metrics agent to forward all metrics to my other Talos cluster that has a storage server running. For the backup itself, I am running a basic cronjob with rsnapshot, a wrapper tool on to of Rsync that uses Linux hard links for its ‘incremental’ backups. The folder structure looks like full-sized backups each day, but the hard links keep the storage usage low. Encryption is provided at the drive level using LUKS2. Again this just uses basic Linux components, so if I need to I can just plug the drive into my Linux laptop and recover all my files without any special tools. The transfer is pull based happens over SSH with read only permission. This makes it so my data stays safe (at least 1 copy) even if one of my servers gets compromised. And last, I am running a small DaemonSet with hdparm that makes sure the drive is configured correctly to spin down when idle. The ZimaBlade is passively cooled so the system is fully silent once the disk spins down.

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: versitygw-s3 namespace: default spec: accessModes: - ReadWriteOnce storageClassName: smb resources: requests: storage: 30Gi

My first bare metal Kubernetes cluster is finally online. It took a while and I tried way to many different things but I eventually ended up with Talos and Omni for the management interface.

My first plan was some fancy net boot setup with IPXE and a custom http/tftp server that managed custom configs for each server. That will install K3s onto MicroOS and join the cluster without ever attaching a keyboard to the server. This was all done with Ignition and Combustion scripts. It worked but was error prone and instable. And later I discovered a very similar project already existed called Matchbox. This uses CoreOS instead of MicroOS, which is almost the same but Fedora flavoured. On top of this K3s is not that simple to setup, its lightweight but not simple. So I was reinventing a shitty wheel. But to my credit, it did work.

Something similar but with NixOS was my 3rd plan but never got to it but I don’t think it would have worked that much better. A bit cleaner but still clunky.

omni venom cluster dashboard view

So going back to Talos OS, which I underestimated at first. I thought it would be to frigid and require a lot of config. It does require some config but it is fully declarative so that was fine. But I was placentally surprised by the headless install via the http API. The install was also fast and as light as MicroOS + K3s. But still the CLI seemed error prone to me and bootstrapping everything was still a lot of manual work.

That is where Omni fills the gap. It was a pain to setup up with all the endpoints and certs that it requires (it also requires some form of SSO). But once that was done it was smooth sailing. You just create the installation media in the web interface and download the ISO (or even just copy over the PXE config in my case). And this setup is not specific for one node. You can use the same IMG on all the nodes and they will connect them self to the Omni server via a Wireguard tunnel waiting on you to make the full install via the UI. Once all nodes connected themselves to my Omni instance I just had to click ‘create cluster’. And once nodes are in the system I can reconfigure (clear, remove/add to a cluster, update…) as much as I want needing a new PXE boot or a fresh ISO. And it can handle many clusters and even automatically setup Wireguard networking in between nodes for a hybrid setup between the cloud and on-prem. It also has native support of Hetzner which ill will servantly test out. The only downside is that Omni is not free for production use. But for homelab it’s perfect (up to now).

Hardware is ‘done’ now, next step: lots of yaml’s.

Resource consumption