July 12, 2020

K8s on Hetzner Cloud

I was looking at having a long-running k8s cluster for testing and pet projects, Hetzner Cloud looks like a good initiative.

Goal

Have a running k8s cluster that fits the following criteria:

  • Secure - the cluster will be exposed to the internet
  • Scalable - I’m not sure what I will run on it, probably a lot of side projects and data mining projects, so I would like to scale it from time to time. At the same time scalability (elasticity) helps ensure the cluster is not fragile
  • Replicable - I want to spin a new cluster, destroy this one and recreate it at will. Replicability is partially related to scalability/elasticity but also about bootstrapping the cluster

Also, a few restrictions (side quests) needed for this:

  • learn more about security and k8s internals
  • don’t spend a lot of money
  • going for a managed solutions is cheating (for now)

Why Hetzner Cloud

I already have experience with EKS and K8s on DigitalOcean, but they can get a bit expensive (especially if you start using other services).

There is a good post Affordable Kubernetes Cluster by Remko Seelig about using GKE with preemptible nodes, but the cluster does not have a lot of computing power.

I already looked at Kubernetes The Hard Way and remembered that Hetzner Cloud offers pretty cheap compute instances (starts at 2,89 €). Working with Hetzner Cloud means that I have to run control plane myself, this may cut in my processing power but has the advantage that I should get a more in-depth look at how it works.

Cloud ProviderControl Plane costworker nodes (8vCPU 16GB ram)worker nodes countMin CostMax Cost
Google (gke)0 (for single-zone or multi-zonal) $0.10 per hour (± 73$ per month) for regional cluster31.86$* -> 105$ ~ 9 * g1-small * preemptible price932$105$
AWS (EKS)$0.10 per hour (± 73$ per month)28$* -> 146$ ~ 4 * a1.large in Ohio * spot insance price491$219$
Azure (AKS)0$26* -> $265.72 ~ 4 * A2 v2 in East US2 * for maximum spot instance discount426$265$
Linode0$80 ~ 4 * (2vCPU 4GB ram)480$80$
Digital Ocean0$80 ~ 4 * (2vCPU 4GB ram)480$80$
Hetzner Cloud5.7€ (assuming CX21 have enough resources)± 22 € (4 CX21)427.7€ (around 32$)27.7€ (around 32$)

There are a few caveats with each solution:

  • preemptible/spot instances can be shut down; we can run part of the instances as normal ones and have smart auto-scaling to provision more if needed
  • spot instance costs can vary for AWS and Azure (preemptible instances have a fixed price on Google)
  • with cloud providers (Google, AWS, Azure, DigitalOcean, Linode) other costs can occur (you can avoid them if you really want):
    • extra storage costs (PV’s for example)
    • data transfer/networking costs
    • backup costs
    • logging costs (for example shipping logs to CloudWatch)
    • load balancer costs
  • GKE with g1-small may not be worth it as the instance is pretty underpowered
  • Running the control plane can be tricky (this is why most of the cloud providers offer managed control plane)

One big advantage of running the control plane and doing everything from scratch is the learning process (one of the side quests for this pet project). Hetzner Cloud has a simple offering, I hope to get support (at least beta) for multiple things:

Existing resources and approaches

Looking online there is already some work done in this direction

There are also multiple installers for k8s; we can even consider k3s or microk8s

Next Steps

I’ll probably look more at cluster api and try to figure out how it works