One of my Kubernetes nodes recently went down without me noticing (issue with VPS provider, not my fault). It was nice that all my services stayed online without any hiccups. But I would like to know in the future when my infra goes down. So time to get the classic stack online Grafana (now with build in Alertmanager), Node Expoter, CAdvisor, kube-state-metrics but no Prometheus this time. I am running VictoriaMetrics instead. It is a drop in replacement for Prometheus and is still written in Go but it is made to be more cost effective. It uses 7 times less memory, CPU and storage then Prometheus. And from my testing those numbers seem to be right. Currently I am storing 28.6 billion datapoints using just 4.37 GiB of storage. Below you can see the stats (for everything monitoring) with a current ingest rate at around 20K rec/s.

Resource consumption