Layer 0 - infra

Create 9 machines on your favorite place. We’ll not details the steps here. We use hetzner cloud. Deploy a debian or ubuntu or your favorite distrib.

Layer 1 - network

Then we need to prepare the Network:


In term of network topology, we’ll use 3 networks, all secured by wireguard:

  • management
  • Ceph backend
  • Kubernetes overlay

Management will be used by kubernetes and ceph components.

The ceph backend will be used for data heavy part of ceph.

And the overlay to allow services running on top of kubernetes to discuss together.

In this step, we’ll only install the 2 first one. The overlay will be installed later.

Layer 2 - kubespray

Use kubespray to deploy your components. It is a set of ansible roles to deploy an HA kubernetes cluster.

Or deploy them by hand. We currently use kubespray, but think of not using it for the next iteration.

We personnaly use the following:

  • containerd - It is the container runtime. Nowadays, the industry is moving from Docker to containerd, so do we.
  • kubeadm - It is developed upstream under kubernetes namespace. It is aimed at becoming the defacto installation tool for kubernetes.
  • canal with wireguard - it is the Kubernetes overlay network.
  • nginx-ingress
  • cert-manager

Layer 3 - storage

Storage with ceph (Or rook if you are bold ;), it is almost production ready)

Layer 4 - backing services

This term is a reference to 12 factor app, and it is not a coincidence! (If you don’t know it, get there and read it!)

This is our curated list of operators:

Layer 5 - upstream operators

Layer 6 - operator

Layer 7 - web UI

UI to let people self serve. The status of this is clearly vaporware, but we hope to release an alpha by begining of 2021 (Except if you have skills and want to work on it already, this would be amazing ;) ).