Andrii Holovin – Blog |

Deploying an HA Kubernetes cluster with an external etcd topology on a local machine

2026-01-12T06:30:00+00:00

In this guide, we will explore the steps to deploy a High Availability (HA) Kubernetes cluster with an external etcd topology on a local machine running macOS. We will use Multipass to create virtual machines, cloud-init for their initialization, kubeadm for cluster initialization, HAProxy as a load balancer for control plane nodes, Calico as a Container Network Interface (CNI), and MetalLB for load balancing traffic to worker nodes.

This guide is intended both for self-study and for taking the first steps in deploying a production-level cluster. Each step is explained in detail, with descriptions of components and their roles.

Cluster Architecture

Highly available Kubernetes clusters are used in production environments to ensure the continuous operation of applications. Redundancy of key cluster components allows avoiding downtime in case of failure of individual control plane nodes or etcd.

Components

We will deploy the cluster using kubeadm on virtual machines running Ubuntu, created by Multipass and initialized using cloud-init.

Multipass — is a tool for quickly creating Ubuntu virtual machines in a cloud-like fashion on Linux, macOS, and Windows. It provides a simple yet powerful command-line interface that allows you to quickly access Ubuntu or create your own local mini-cloud.

Local development and testing can be challenging, but Multipass simplifies these processes by automating the deployment and teardown of infrastructure. Multipass has a library of ready-to-use images that can be used to launch specialized virtual machines or your own custom virtual machines, configured using the powerful cloud-init interface.

Developers can use Multipass to prototype cloud deployments and create new, customized Linux development environments on any machine. Multipass is the fastest way for Mac and Windows users to get an Ubuntu command line in their systems. You can also use it as a sandbox to try new things without affecting the host machine and without the need for dual-booting.
Cloud-init — is an industry-standard method of initialization, present in many distributions, for creating cloud instances on various platforms. It is supported by all major public cloud service providers, private cloud infrastructure providers, and bare-metal installations.

During boot, cloud-init detects the cloud it is running in and initializes the system accordingly. Cloud instances are automatically provided with networking, storage, SSH keys, packages, and other pre-configured system components on first boot.

Cloud-init provides the necessary linkage between the cloud instance startup and its connection so that it works as expected.

For cloud users, cloud-init provides cloud instance configuration management at first boot without the need to manually install required components. For cloud service providers, it provides instance initialization that can be integrated with your cloud.

If you want to learn more about what cloud-init is, why it is needed, and how it works, read the detailed description in its documentation.
Kubeadm — is a tool designed to provide the kubeadm init and kubeadm join commands as the best “quick practical ways” to create Kubernetes clusters.

Kubeadm performs the necessary actions to get a minimal viable cluster up and running. By design, it only deals with the cluster deployment process, creating machine instances is out of its scope. Installing various additional components, such as the Kubernetes Dashboard, monitoring tools, and cloud-specific overlays, is also not part of its tasks.
Etcd with external topology. Etcd is a distributed key-value store with high consistency that provides a reliable way to store data that needs to be accessed by a distributed system or a cluster of machines. It correctly handles leader elections during network partitions and can withstand machine failures, even leader nodes.

Kubernetes uses etcd to store all of its configuration and cluster state. This includes information about nodes, pods, network configuration, secrets, and other Kubernetes resources. The high-availability topology of a Kubernetes cluster involves two placement options for etcd: stacked etcd topology and external etcd topology. In this guide, we will focus on the external etcd topology, where etcd is deployed separately from the Kubernetes control plane nodes. This provides better isolation, scalability, and flexibility in managing the cluster.
HAProxy. To distribute traffic to the control plane nodes, we will use the load balancer HAProxy. This will allow us to ensure high availability of the Kubernetes API server. In addition, we will deploy local HAProxy instances on each control plane node to access the etcd cluster nodes.
Calico. Since kubeadm does not create the network in which pods operate, we will use Calico, a popular tool that implements the Container Network Interface (CNI) and provides network policies. It is a unified platform for all Kubernetes networking, network security, and observability needs, working with any Kubernetes distribution. Calico simplifies network security enforcement using network policies.

Cluster Topology

graph TB
  subgraph C [Control Plane and etcd]
    direction TB
    subgraph CP [Control Plane Nodes]
        direction LR
        CP1 --> CP2
        CP2 --> CP1
        CP1 --> CP3
        CP3 --> CP2
        CP2 --> CP3
        CP3 --> CP1
    end

    subgraph E [etcd cluster]
        direction LR
        E1 --> E2
        E2 --> E1
        E1 --> E3
        E3 --> E2
        E2 --> E3
        E3 --> E1
    end

    subgraph HA [Keepalived
    10.10.0.100]
      HA1(HAProxy
      10.10.0.101) <----->
      HA2(HAProxy
      10.10.0.102)
    end

    CP <--> HA
    E <--> CP

  end

    W1(Worker
    Node 1
    10.10.0.31)
    W2(Worker
    Node 2
    10.10.0.32)
    W3(Worker
    Node 3
    10.10.0.33)
    WN(Worker
    Node …
    10.10.0.9X)

    HA --> PN(Calico CNI)
    PN --> W1 & W2 & W3 & WN--> WLB <--> U(Users)

    CP1(Control Plane
    Node 1
    10.10.0.21)
    CP2(Control Plane
    Node 2
    10.10.0.22)
    CP3(Control Plane
    Node 3
    10.10.0.23)
    E1(etcd
    Node 1
    10.10.0.11)
    E2(etcd
    Node 2
    10.10.0.12)
    E3(etcd
    Node 3
    10.10.0.13)

    WLB(MetalLB
    10.10.0.200)

Prerequisites

To deploy the cluster, we will need:

A local computer with Multipass installed. For macOS, it can be installed using brew install multipass
kubectl — The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs (brew install kubectl)
Basic knowledge of Kubernetes and Linux
yq (mikefarah’s version) — a lightweight and portable command-line processor for YAML, JSON, INI, and XML.

SSH Key Setup

Secure Shell, SSH (English: Secure Shell) is a network protocol that allows remote management of a computer. It encrypts all traffic, including the authentication process and the transmission of passwords and secrets. To access the virtual machines, you need to set up SSH keys. We will use these keys to connect to all created virtual machines and transfer files between them.

Generating an SSH Key

To generate a key pair (public and private), run the following command:

# Create a new pair of SSH keys (if you don't have one)
ssh-keygen -t rsa -b 4096 -C "k8s-cluster" -f ~/.ssh/k8s_cluster_key

# View the public key
cat ~/.ssh/k8s_cluster_key.pub

Updating cloud-init Configuration

Use the obtained public key in the snipets/cloud-init-users.yaml file in the ssh-authorized-keys section:

users:
  - name: k8sadmin
    ssh_authorized_keys:
      # Replace with your actual public key
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...

You can also use variable substitution at runtime

$ multipass launch --cloud-init - <<EOF
users:
  - name: username
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - $( cat ~/.ssh/id_rsa.pub )
EOF

Accessing Virtual Machines via SSH

After deployment, we can access the virtual machines using the following commands:

multipass shell <vm-name>
# or
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@<vm-ip>

Multipass has a built-in shell command that allows us to connect to any created virtual machine without additional SSH configuration. However, it uses the default user ubuntu. If you want to connect as the k8sadmin user, which we will create using cloud-init, use ssh -i ~/.ssh/k8s_cluster_key k8sadmin@<vm-ip>.

Environment Preparation

Create a project directory and the necessary files.

mkdir k8s-ha-cluster
cd k8s-ha-cluster

Creating Scripts and Configurations

Virtual Machine Parameters

To deploy the virtual machines, we need the following parameters:

Node	Quantity	`cpus`	`memory`	`disk`	`network`
etcd	3+ or (2n+1)^[1]	2	2G	5G	name=en0,mode=manual
control plane	3	2	2.5G	8G	name=en0,mode=manual
worker nodes	3+	2	2G	10G	name=en0,mode=manual
HAProxy+Keepalived	2	1	1G	5G	name=en0,mode=manual

Cloud-init Configurations

We will use pre-created cloud-init virtual machine configuration snippets to speed up their deployment.

User Creation

Let’s create the user k8sadmin settings in snipets/cloud-init-user.yaml:

# Create the directory if it doesn't exist
mkdir -p snipets

cat << EOF > snipets/cloud-init-user.yaml
# User settings
users:
  - name: k8sadmin
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    groups: sudo,users,containerd
    homedir: /home/k8sadmin
    shell: /bin/bash
    lock_passwd: false
    ssh_authorized_keys:
      # Your SSH key
      - $( cat ~/.ssh/k8s_cluster_key.pub )
EOF

We have already discussed the user configuration in the SSH key settings. You can read about other parameters in the Including users and groups section of the cloud-init documentation.

Base Configuration for Cluster Nodes

The base cloud-init configuration for our cluster nodes will be in snipets/cloud-init-base.yaml:

#cloud-config

# Add user settings from snipets/cloud-init-users.yaml here

# Commands that run early in the boot stage to prepare the GPG key
bootcmd:
  - mkdir -p /etc/apt/keyrings
  - curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Apt repository configuration
apt:
  sources:
    kubernetes.list:
      source: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /"

# System update
package_update: true
package_upgrade: true

To install the components necessary for the cluster operation, we need to use the Kubernetes project repository. In the bootcmd section, which is very similar to runcmd, but the commands from which are executed at the very beginning of the boot process, we obtain the GPG key and save it to /etc/apt/keyrings, and then add the Kubernetes repository to the apt sources list in the apt section. The package_update and package_upgrade lines ensure that the latest system updates are obtained during boot, equivalent to running apt-get update and apt-get upgrade.

In the packages section, we will specify all the necessary packages to be installed on each virtual machine of the cluster.

# Install basic packages
packages:
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg
  - lsb-release
  - containerd
  - kubelet
  - kubeadm
  - kubectl # Except for worker nodes

In addition to system service packages, we also install containerd, kubelet, kubeadm, and kubectl, which are the core components for the operation of the Kubernetes cluster.

containerd is the container runtime engine used by Kubernetes to run containerized applications. kubelet is the agent that runs on each node in the Kubernetes cluster and is responsible for running containers in pods. kubeadm is the tool for quickly deploying the Kubernetes cluster, and kubectl is the command-line tool for interacting with the Kubernetes cluster.

# User configuration
users:
  - name: k8sadmin
    groups: sudo,users,containerd

Our user is a member of the containerd group, which we will use to access the /run/containerd/containerd.sock socket, enabling the use of crictl without the need to use sudo.

The write_files instruction in cloud-init allows creating files with specified content during the initialization of the virtual machine. We will use it to create files for configuring kernel modules for Kubernetes operation, enabling IP forwarding, creating settings for crictl, and a script that we will use to initialize and start kubelet. (See the write_files section in the cloud-init-base.yaml file)

After we have created all the necessary files using write_files, we can execute the specified commands for system configuration in the runcmd section. Here we can specify the commands that we would have to execute manually after the first system boot. In our case, we will freez the versions of kubeadm, kubelet, and kubectl, disable swap, create a configuration file for containerd, and start it, as well as enable and start the kubelet service. (See the runcmd section in the cloud-init-base.yaml file)

Create two files:

snipets/cloud-init-config.yaml — to assign a static IP address to our virtual machines

cat << EOF > snipets/cloud-init-config.yaml
#cloud-config
timezone: Europe/Kyiv

write_files:
  # Assigning a static IP address
  # Variant with alias for bridge101 for the second network interface
  - path: /etc/netplan/60-static-ip.yaml
    # Explicitly specify the value type
    # permissions: !!str "0755" # https://github.com/canonical/multipass/issues/4176
    permissions: !!str '0600'
    content: |
      network:
        version: 2
        ethernets:
          enp0s2:
            # Here you specify the IP address of the specific virtual machine
            addresses:
              # - 10.10.0.24/24
            routes:
              - to: 10.10.0.0/24
                scope: link
runcmd:
  # Applying settings for using a static IP address
  - netplan apply

  # Setting vim as the default editor (optional)
  - update-alternatives --set editor /usr/bin/vim.basic
EOF

☝️ You can also specify that you want to use vim as your default editor here. If you prefer nano, comment out or remove the line - update-alternatives --set editor /usr/bin/vim.basic

and, the file snipets/cloud-init-base.yaml with basic settings for the cluster virtual machines

cat << 'EOF' > snipets/cloud-init-base.yaml
# Runs at an early boot stage to prepare the GPG key
bootcmd:
  - mkdir -p /etc/apt/keyrings
  - curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Apt repository configuration
apt:
  sources:
    kubernetes.list:
      source: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /"

# System update
package_update: true
package_upgrade: true

# Install basic packages
packages:
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg
  - lsb-release
  - containerd
  - kubelet
  - kubeadm
  - kubectl

write_files:
  # Kernel module settings for Kubernetes
  - path: /etc/modules-load.d/k8s.conf
    # permissions: !!str '0644'
    content: |
      overlay
      br_netfilter

  # Enabling IPv4 packet forwarding
  # https://andygol-k8s.netlify.app/docs/setup/production-environment/container-runtimes/#prerequisite-ipv4-forwarding-optional
  - path: /etc/sysctl.d/k8s.conf
    # permissions: !!str '0644'
    content: |
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1

  # Config for crictl
  # https://github.com/containerd/containerd/blob/main/docs/cri/crictl.md#install-crictl
  # https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md
  - path: /etc/crictl.yaml
    # permissions: !!str '0644'
    content: |
      runtime-endpoint: unix:///run/containerd/containerd.sock
      image-endpoint: unix:///run/containerd/containerd.sock
      timeout: 10
      debug: false

  # Setting group for containerd socket
  - path: /etc/systemd/system/containerd.service.d/override.conf
    content: |
      [Service]
      ExecStartPost=/bin/sh -c "chgrp containerd /run/containerd/containerd.sock && chmod 660 /run/containerd/containerd.sock"

  # Script to start kubelet
  - path: /usr/local/bin/kubelet-start.sh
    permissions: !!str "0755"
    content: |
      #!/bin/bash

      echo "Starting kubelet service and waiting for readiness (timeout 300s)..."

      # Enable and start the service
      sudo systemctl enable --now kubelet

      WAIT_LIMIT=300       # Maximum wait time in seconds
      ELAPSED_TIME=0       # Time already passed
      SLEEP_INTERVAL=1     # Initial interval (1 second)

      while ! systemctl is-active --quiet kubelet; do
          if [ "$ELAPSED_TIME" -ge "$WAIT_LIMIT" ]; then
              echo "--------------------------------------------------------------" >&2
              echo "BROKEN-DOWN: kubelet did not start within $WAIT_LIMIT seconds." >&2
              echo "Last error logs:"                                               >&2
              journalctl -u kubelet -n 20 --no-pager                                >&2
              echo "--------------------------------------------------------------" >&2
              exit 1
          fi

          echo "Waiting for kubelet... (elapsed $ELAPSED_TIME/$WAIT_LIMIT sec,"
          echo "next attempt in ${SLEEP_INTERVAL}sec)"

          sleep $SLEEP_INTERVAL

          # Update counters
          ELAPSED_TIME=$((ELAPSED_TIME + SLEEP_INTERVAL))

          # Double the interval for the next time (progressive)
          # But do not make the interval longer than 20 seconds, to not "oversleep" readiness
          SLEEP_INTERVAL=$((SLEEP_INTERVAL * 2))
          if [ "$SLEEP_INTERVAL" -gt 20 ]; then
              SLEEP_INTERVAL=20
          fi
      done

      echo "Kubelet successfully started in $ELAPSED_TIME seconds. Continuing..."

runcmd:
  # Freezing the versions of Kubernetes packages
  - apt-mark hold kubelet kubeadm kubectl

  # Loading kernel modules
  - modprobe overlay
  - modprobe br_netfilter

  # Applying sysctl parameters
  - sysctl --system

  # Configuring containerd
  #
  # Setting the systemd cgroup driver
  # https://andygol-k8s.netlify.app/docs/setup/production-environment/container-runtimes/#containerd-systemd
  # https://github.com/containerd/containerd/blob/main/docs/cri/config.md#cgroup-drivercrictl pull
  #
  # Overriding the pause image
  # https://andygol-k8s.netlify.app/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

  - mkdir -p /etc/containerd
  - containerd config default | tee /etc/containerd/config.toml
  - sed -i 's|SystemdCgroup = false|SystemdCgroup = true|g; s|sandbox_image = "registry.k8s.io/pause.*"|sandbox_image = "registry.k8s.io/pause:3.10.1"|' /etc/containerd/config.toml
  - [ systemctl, daemon-reload ]
  - [ systemctl, restart, containerd ]
  - [ systemctl, enable, containerd ]

  # Disabling swap
  # https://andygol-k8s.netlify.app/docs/concepts/cluster-administration/swap-memory-management/#swap-and-control-plane-nodes
  # https://andygol-k8s.netlify.app/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#swap-configuration
  - swapoff -a
  - sed -i '/ swap / s/^/#/' /etc/fstab

  # Enabling kubelet
  - /usr/local/bin/kubelet-start.sh
EOF

The default access rights for files created by cloud-init are 0644. If you want to specify them explicitly, uncomment the corresponding lines # permissions: !!str ‘0644’. We specify the data type explicitly (!!str ‘0644’) due to the issue described in ticket # 4176.

Configuration for etcd nodes

For etcd nodes, we will use the base configuration with additional settings specific to etcd. Create the file snipets/cloud-init-etcd.yaml:

# Extend cloud-init-base.yaml with the following settings

cat << EOF > snipets/cloud-init-etcd.yaml
write_files:
  # https://andygol-k8s.netlify.app/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setup-up-the-cluster
  - path: /etc/systemd/system/kubelet.service.d/kubelet.conf
    content: |
      apiVersion: kubelet.config.k8s.io/v1beta1
      kind: KubeletConfiguration
      authentication:
        anonymous:
          enabled: false
        webhook:
          enabled: false
      authorization:
        mode: AlwaysAllow
      cgroupDriver: systemd
      address: 127.0.0.1
      containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
      staticPodPath: /etc/kubernetes/manifests

  - path: /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/usr/bin/kubelet --config=/etc/systemd/system/kubelet.service.d/kubelet.conf
      Restart=always
EOF

We will use the basic settings we created earlier and add etcd-specific settings in the write_files section, where we create a configuration file for kubelet that configures it to work with etcd, as described in the section “Configuring a highly available etcd cluster with kubeadm” section of the Kubernetes documentation.

Configuring the HAProxy balancer node

For the control panel traffic load balancer nodes, we will create the file snipets/cloud-init-haproxy.yaml, which will install the haproxy and keepalived packages from the standard repository and add the settings. To create a virtual machine for HAProxy, we will use the basic settings from configs/cloud-init-user.yaml and extend them with write_files: instructions to create the balancer configuration files — /etc/haproxy/haproxy.cfg, /etc/keepalived/keepalived.conf. (See the section “Configuring the load balancer for control plane nodes (HAProxy+Keepalived).”)

Choosing a network topology

For this demonstration, we will choose a compact network. (10.10.0.0/24)

10.10.0.0/24      - Main subnet (256 addresses)
├─ 10.10.0.1      - Gateway/Bridge (on the host machine)
├─ 10.10.0.10-19  - etcd (3+ nodes)
├─ 10.10.0.20-29  - Control Plane (3+ masters)
├─ 10.10.0.30-50  - Workers (up to 20 workers)
└─ 10.10.0.100    - HAProxy/Keepalived (two nodes 10.10.0.101/10.10.0.102)

To assign static addresses to Multipass virtual machines, we will use one of the options described in the article “Adding a static IP address to Multipass virtual machines on macOS”.

Add the appropriate section to the cloud-init file with the static network address configuration on the second network interface (see snipets/cloud-init-config.yaml).

Creating an etcd cluster

Let’s start by creating an etcd cluster, where our highly available Kubernetes cluster will store the configuration and desired state of system objects.

Deploying the first node of the etcd cluster

Let’s start deploying our cluster by creating the first etcd node.

export VM_IP="10.10.0.11/24"

multipass launch --name ext-etcd-1 \
  --cpus 2 --memory 2G --disk 5G \
  --network name=en0,mode=manual \
  --cloud-init <( yq eval-all '
      # Merge all files into a single object
      . as $item ireduce ({}; . *+ $item) |

      # Remove kubectl from the list of packages
      del(.packages[] | select(. == “kubectl”)) |

      # Update the network configuration
      with(.write_files[] | select(.path == “/etc/netplan/60-static-ip.yaml”);
        .content |= (
          from_yaml |
          .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] |
          to_yaml
        )
      ) ' \
      snipets/cloud-init-config.yaml \
      snipets/cloud-init-user.yaml \
      snipets/cloud-init-base.yaml \
      snipets/cloud-init-etcd.yaml )

The command multipass launch --name ext-etcd-1 will start deploying a virtual machine with the name specified in the --name/-n parameter, in this case ext-etcd-1; The parameters --cpus 2 --memory 2G --disk 5G specify the number of processor cores, memory, and disk space, respectively; --network name=en0,mode=manual will create another network interface for the virtual machine, whose IP address will be assigned via the VM_IP variable.

Since etcd actively writes data to disk, database performance directly depends on the performance of the disk subsystem. For production use, SSD storage systems are strongly recommended. The minimum disk space should be at least 2 GB by default. Accordingly, to avoid placing data in swap, the amount of RAM should cover this quota. 8 GB is the recommended maximum for typical deployments. Requirements for etcd machines for small industrial clusters: 2 vCPUs, 8 GB of memory, and 50-80 GB SSD. (See https://andygol-etcd.netlify.app/docs/v3.5/op-guide/hardware/#small-cluster, https://andygol-etcd.netlify.app/docs/v3.5/faq/#system-requirements)

We will merge cloud-init parameters on the fly by combining our template files snipets/cloud-init-config.yaml, snipets/cloud-init-user.yaml, snipets/cloud-init-base.yaml, snipets/cloud-init-etcd.yaml using yq.

If you did not create a temporary virtual machine to initialize bridge101 and did not add an alias for it, after deploying the virtual machine, it’s time to do the following. Run the following on your host:

# Define the name of the bridge. Most likely, the name will be bridge101.
ifconfig -v | grep -B 20 “member: vmenet” | grep “bridge” | awk -F: ‘{print $1}’ | tail -n 1

# Add the address
sudo ifconfig bridge101 10.10.0.1/24 alias

# Check that it has been added
ifconfig bridge101 | grep “inet ”

If you did this 👆 after creating a temporary virtual machine, you can now delete it

multipass delete <temp-vm> --purge

Configuring the first etcd node

Let’s log into our node using SSH.

ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.11

Since we don’t have any CA certificates yet, we need to generate them.

sudo kubeadm init phase certs etcd-ca

We will get two files, ca.crt and ca.key, in the /etc/kubernetes/pki/etcd/ folder.

Now let’s create a configuration file for kubeadm kubeadmcfg.yaml using the appropriate values in the variables ETCD_HOST (IP address of the virtual machine) and ETCD_NAME (its short name). Note the value of the ETCD_INITIAL_CLUSTER_STATE variable, which indicates that we are creating a new etcd cluster. We will add other nodes to it later.

ETCD_HOST=$(hostname -I | awk '{print $2}')
ETCD_NAME=$(hostname -a)
ETCD_INITIAL_CLUSTER=${ETCD_INITIAL_CLUSTER:-"${ETCD_NAME}=https://${ETCD_HOST}:2380"}
ETCD_INITIAL_CLUSTER_STATE=${ETCD_INITIAL_CLUSTER_STATE:-"new"}
cat << EOF > $HOME/kubeadmcfg.yaml
---
apiVersion: "kubeadm.k8s.io/v1beta4"
kind: InitConfiguration
nodeRegistration:
    name: ${ETCD_NAME}
localAPIEndpoint:
    advertiseAddress: ${ETCD_HOST}
---
apiVersion: "kubeadm.k8s.io/v1beta4"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${ETCD_HOST}"
        peerCertSANs:
        - "${ETCD_HOST}"
        extraArgs:
        - name: initial-cluster
          value: ${ETCD_INITIAL_CLUSTER}
        - name: initial-cluster-state
          value: ${ETCD_INITIAL_CLUSTER_STATE}
        - name: name
          value: ${ETCD_NAME}
        - name: listen-peer-urls
          value: https://${ETCD_HOST}:2380
        - name: listen-client-urls
          value: https://${ETCD_HOST}:2379,https://127.0.0.1:2379
        - name: advertise-client-urls
          value: https://${ETCD_HOST}:2379
        - name: initial-advertise-peer-urls
          value: https://${ETCD_HOST}:2380
EOF

Using the created kubeadmcfg.yaml file, which we placed in the home directory of the k8sadmin user, we will generate etcd certificates and create a static pod manifest for the etcd cluster node.

# 1. Certificate issuance
sudo kubeadm init phase certs etcd-server --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs etcd-peer --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs etcd-healthcheck-client --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs apiserver-etcd-client --config=$HOME/kubeadmcfg.yaml

Now we should have the following keys and certificates available

/home/k8sadmin
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── ca.key
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

After creating the appropriate certificates, it’s time to create a static pod manifest. As a result, we should have a file /etc/kubernetes/manifests/etcd.yaml.

# 2. Creating a static pod manifest
sudo kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml

Once we have the /etc/kubernetes/manifests/etcd.yaml manifest, the node’s kubelet should pick it up, download the container image, and start the pod with etcd, after which the first node of our cluster should respond to health checks.

crictl exec $(crictl ps --label io.kubernetes.container.name=etcd --quiet) etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://10.10.0.11:2379 \
   endpoint health -w table

+-------------------------+--------+------------+-------+
|        ENDPOINT         | HEALTH |    TOOK    | ERROR |
+-------------------------+--------+------------+-------+
| https://10.10.0.11:2379 |   true | 7.777048ms |       |
+-------------------------+--------+------------+-------+

Deploying subsequent etcd cluster nodes

To deploy the next etcd cluster nodes: ext-etcd-2, ext-etcd-3, and so on (as needed), change the value of VM_IP to “10.10.0.12/24” and the virtual machine name in the --name parameter to ext-etcd-2, respectively.

export VM_IP="10.10.0.12/24"

multipass launch --name ext-etcd-2 \
  --cpus 2 --memory 2G --disk 5G \
  --network name=en0,mode=manual \
  --cloud-init <( yq eval-all '
      # Merge all files into a single object
      . as $item ireduce ({}; . *+ $item) |

      # Remove kubectl from the package list
      del(.packages[] | select(. == “kubectl”)) |

      # Update network configuration
      with(.write_files[] | select(.path == “/etc/netplan/60-static-ip.yaml”);
        .content |= (
          from_yaml |
          .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] |
          to_yaml
        )
      ) ' \
      snipets/cloud-init-config.yaml \
      snipets/cloud-init-user.yaml \
      snipets/cloud-init-base.yaml \
      snipets/cloud-init-etcd.yaml )

After completing the deployment of the node, let’s check if we have SSH access to it:

ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.12 -- ls -la

Configuring etcd nodes and joining them to the cluster

On our etcd node ext-etcd-1, which is already running, we will execute the following command to get instructions for joining the node ext-etcd-2 to the etcd cluster. After the member add command, we will specify the name of the node to be joined and the path to it in the --peer-urls parameter.

crictl exec $(crictl ps --label io.kubernetes.container.name=etcd --quiet) etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://10.10.0.11:2379 \
   member add ext-etcd-2 --peer-urls=https://10.10.0.12:2380

etcdctl will register a new member of the etcd cluster, and in response we will receive its ID and the string “ext-etcd-1=https://10.10.0.11:2380,ext-etcd-2=https://10.10.0.12:2380” with a complete list of cluster nodes (including the new member).

Member e3e9330902f761c3 added to cluster 3f0c3972eda275cb

ETCD_NAME="ext-etcd-2"
ETCD_INITIAL_CLUSTER="ext-etcd-1=https://10.10.0.11:2380,ext-etcd-2=https://10.10.0.12:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.0.12:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Creating kubeadmcfg.yaml

Create the configuration file ~/kubeadmcfg.yaml on the ext-etcd-2 node, replacing the variable values with those you just obtained after executing the … member add … command.

The next mandatory step is to copy the CA files from ext-etcd-1 to ext-etcd-2. For convenience, the steps for transferring CA files have been combined into a script that you need to create on your host machine.

cat << ‘EOF’ > copy-etcd-ca.sh
#!/bin/bash

# --- Default settings (change as needed) ---
DEFAULT_KEY="~/.ssh/k8s_cluster_key"
DEFAULT_SRC="10.10.0.11"
DEFAULT_DEST=“10.10.0.12”
USER="k8sadmin"
CERT_PATH="/etc/kubernetes/pki/etcd"

# --- Assigning arguments ---
# $1 - first argument (source host), $2 - second (destination host), $3 - third (path to key)
SRC_HOST=${1:-$DEFAULT_SRC}
DEST_HOST=${2:-$DEFAULT_DEST}
KEY=${3:-$DEFAULT_KEY}

echo “Parameters used:”
echo “  Source:      ${SRC_HOST}”
echo “  Destination: ${DEST_HOST}”
echo “  SSH Key:     ${KEY}”
echo “---------------------------------------”

# 1. Preparing files on the source
echo “[1/4] Preparing files on ${SRC_HOST}...”
ssh -i “${KEY}” “${USER}@${SRC_HOST}” “sudo cp $CERT_PATH/ca.* /tmp/ && sudo chown ${USER}:${USER} /tmp/ca.*” || exit 1

# 2. Transferring files between hosts
echo “[2/4] Copying from ${SRC_HOST} to $DEST_HOST...”
scp -i “${KEY}” “${USER}@${SRC_HOST}:/tmp/ca.*” “${USER}@$DEST_HOST:/tmp/” || exit 1

# 3. Placing files on the target host
echo “[3/4] Placing files on $DEST_HOST...”
ssh -i “${KEY}” “${USER}@$DEST_HOST” "sudo mkdir -p $CERT_PATH && sudo mv /tmp/ca.* $CERT_PATH/ && sudo chown root:root $CERT_PATH/ca.* && sudo chmod 600 $CERT_PATH/ca.key" || exit 1

# 4. Cleanup
echo “[4/4] Deleting temporary files...”
ssh -i “${KEY}” “${USER}@${SRC_HOST}” “rm /tmp/ca.*”

echo “ca.crt and ca.key files successfully transferred from ${SRC_HOST} to $DEST_HOST”
EOF

chmod +x copy-etcd-ca.sh

Transfer the CA files from host 10.10.0.11 to 10.10.0.12 using the command (specify your parameters if necessary):

./сopy-etcd-ca.sh 10.10.0.11 10.10.0.12

Now, let’s generate key and certificate files using the created kubeadmcfg.yaml file on the ext-etcd-2 node.

# 1. Certificate issuance
sudo kubeadm init phase certs etcd-server --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs etcd-peer --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs etcd-healthcheck-client --config=$HOME/kubeadmcfg.yaml
sudo kubeadm init phase certs apiserver-etcd-client --config=$HOME/kubeadmcfg.yaml

After creating the necessary key files and certificates for the second etcd node, delete the file with the CA private key /etc/kubernetes/pki/etcd/ca.key, as it is no longer needed here.

# 2. Delete /etc/kubernetes/pki/etcd/ca.key
sudo rm -f /etc/kubernetes/pki/etcd/ca.key

Now that we have the necessary certificates in place, let’s create a manifest for deploying a static pod.

sudo kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml

In a minute or two, we will review the list of nodes in our etcd cluster.

crictl exec $(crictl ps --label io.kubernetes.container.name=etcd --quiet) etcdctl \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --endpoints https://10.10.0.11:2379  member list -w table

+------------------+---------+------------+-------------------------+-------------------------+------------+
|        ID        | STATUS  |    NAME    |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |
+------------------+---------+------------+-------------------------+-------------------------+------------+
| 86041dd24c0806ff | started | ext-etcd-1 | https://10.10.0.11:2380 | https://10.10.0.11:2379 |      false |
| e3e9330902f761c3 | started | ext-etcd-2 | https://10.10.0.12:2380 | https://10.10.0.12:2379 |      false |
+------------------+---------+------------+-------------------------+-------------------------+------------+

To add the next cluster node, repeat the same steps as for ext-etcd-2. Note that the ETCD_INITIAL_CLUSTER_STATE variable must have the value “existing”, and in the ETCD_INITIAL_CLUSTER variable for creating ~/kubeadmcfg.yaml on the ext-etcd-3 node, you will need to specify all the nodes that are to be members of the cluster. For the ext-etcd-3 node with IP 10.10.0.13, this variable will look like this:

ETCD_INITIAL_CLUSTER="ext-etcd-1=https://10.10.0.11:2380,ext-etcd-2=https://10.10.0.12:2380,ext-etcd-3=https://10.10.0.13:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Canceling node joining to the cluster

If for any reason you do not want to join a node to the etcd cluster, you need to cancel the previous join command. To do this, you need to remove this node from the list of cluster members by its ID. Even if the node is not yet running, it is already registered in the cluster in the unstarted state.

Find the ID of the desired node. It can be found in the first line of the join command output, or by obtaining a list of all cluster members:

crictl exec $(crictl ps --label io.kubernetes.container.name=etcd --quiet) etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://10.10.0.11:2379 \
   member list

In the output, you will see a line that looks something like this:
62f5145363dbf1b5, unstarted, ext-etcd-2, https://10.10.0.12:2380, ...

or in tabular form

+------------------+-----------+------------+-------------------------+-------------------------+------------+
|        ID        |  STATUS   |    NAME    |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |
+------------------+-----------+------------+-------------------------+-------------------------+------------+
| 167ef81a292916d4 |   started | ext-etcd-2 | https://10.10.0.12:2380 | https://10.10.0.12:2379 |      false |
| 62f5145363dbf1b5 | unstarted |            | https://10.10.0.14:2380 |                         |      false |
| 86041dd24c0806ff |   started | ext-etcd-1 | https://10.10.0.11:2380 | https://10.10.0.11:2379 |      false |
| ba9a6c0afb514fec |   started | ext-etcd-3 | https://10.10.0.13:2380 | https://10.10.0.13:2379 |      false |
+------------------+-----------+------------+-------------------------+-------------------------+------------+

Copy the ID (here, 62f5145363dbf1b5) and execute this command to delete it:

etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://10.10.0.11:2379 \
   member remove <ID_ВУЗЛА>

💡 The same applies to removing any node from the list of cluster members. If you want to replace one cluster node with another, first remove the “old” node, then add the new node.

Why is this important?

If you simply leave the node in the unstarted state, etcd will constantly try to contact it, which can lead to increased latency or problems with reaching a quorum in the future.

Tip: Before attempting to rejoin, make sure that the old etcd data directory (data-dir) has been deleted on the new node (10.10.0.12) so that it can start synchronization from scratch as a new member of the cluster.

Removing data-dir

The path to the data directory (data-dir) depends on how etcd is installed (via kubeadm or as a separate service).

If etcd runs as a Static Pod (most common case, kubeadm)

Review the pod manifest on the node where etcd is already running:
```
grep "data-dir" /etc/kubernetes/manifests/etcd.yaml
```
The standard path in this case is usually: /var/lib/etcd
If etcd runs as a system service (Systemd)

If you installed etcd manually or via binary files, check the service configuration:
```
systemctl cat etcd | grep data-dir
```
Or look in the configuration file (if it exists): /etc/etcd/etcd.conf.
Checking via running process information

You can see the path directly in the arguments of the running process:
```
ps -ef | grep etcd | grep data-dir
```

When you delete the false node entry (as described above) and want to try again:

Clear the directory on the node before restarting it

  rm -rf /var/lib/etcd/*

Note: Make sure you are deleting data on the correct node.

Check access permissions: After clearing, make sure that the user running etcd (usually etcd or root) has write permissions for this directory.

Configuring the load balancer for control plane nodes (HAProxy+Keepalived)

In a Kubernetes cluster with HA architecture, the load balancer must be started BEFORE the first control plane node is initialized.

This is the “first brick rule”: we cannot build a wall if we have not determined where it will stand. controlPlaneEndpoint is an entry point that must be accessible from the first second of the cluster’s life.

The sequence of actions we will follow:

Deploy HAProxy+Keepalived (10.10.0.100)
- It is not necessary to have “live” backends (control plane nodes) at this point, but the balancer must listen on port 6443 and be accessible on the network.
Add control plane nodes to the HAProxy config.
- We haven’t run kubeadm init on the first node yet, but we’ll add it and the IPs of other nodes to the HAProxy backends.
Run kubeadm init on the first control plane node.
- When kubeadm tries to “knock” on 10.10.0.100:6443, the balancer will redirect this request to the very first node (where the API server just came up), and the initialization operation will complete successfully.
Let’s join the other Control Plane nodes.
- Let’s use kubeadm join ... --control-plane.

Temporary “hack” (if you can’t set up HAProxy right now)

If you are unable to deploy a separate machine for the balancer right now, you can use an “IP address maneuver”:

Temporarily assign IP 10.10.0.100 to the first Control Plane node as an additional one (via alias).
Run kubeadm init. The system will see “itself” at this address and complete the configuration.
Later, when you deploy the real HAProxy, move this IP there.

Configuring HAProxy+Keepalived

To make our load balancer truly fault-tolerant (High Availability), we need to configure Keepalived. It will allow two HAProxy nodes to share a single “floating” IP address (Virtual IP — VIP).

Netplan configuration

We will leave the address 10.10.0.100 for Keepalived, and in the network settings section of the HAProxy virtual machine (we will have two of them), we will do the following:

  - path: /etc/netplan/60-static-ip.yaml
    permissions: !!str ‘0600’
    content: |
      network:
        version: 2
        ethernets:
          enp0s2:
            addresses:
              - 10.10.0.101/24 # Real IP of node LB1 (for the second one it will be .102)
            routes:
              - to: 10.10.0.0/24
                scope: link

We will specify the address 10.10.0.101 for the primary balancer and 10.10.0.102 for the backup.

Keepalived configuration

Add the following block to write_files. This configuration will force Keepalived to monitor the status of HAProxy and transfer the VIP to another node if the service or machine goes down.

write_files:
  - path: /etc/keepalived/keepalived.conf
    content: |
      vrrp_script check_haproxy {
          script “killall -0 haproxy” # Check if the process is alive
          interval 2
          weight 2
      }

      vrrp_instance VI_1 {
          state MASTER              # On the second node, specify BACKUP
          interface enp0s2          # Name of your interface
          virtual_router_id 51      # Must be the same for both LBs
          priority 101              # On the second node, specify 100
          advert_int 1

          authentication {
              auth_type PASS
              auth_pass k8s_secret  # Shared password
          }

          virtual_ipaddress {
              10.10.0.100/24        # Your VIP address for Cluster Endpoint
          }

          track_script {
              check_haproxy
          }
      }

Kernel configuration (Sysctl)

In order for HAProxy to “sit” on the IP address 10.10.0.100, which does not yet belong to it (until Keepalived raises it), you need to enable nonlocal_bind.

Let’s add this to write_files:

write_files:
  - path: /etc/sysctl.d/99-kubernetes-lb.conf
    content: |
      net.ipv4.ip_nonlocal_bind = 1

And let’s add commands to runcmd for application:

runcmd:
  - sysctl --system
  - netplan apply
  - systemctl enable --now haproxy
  - systemctl enable --now keepalived

How it works together

HAProxy listens on port 6443, but it only “sees” traffic coming to the VIP 10.10.0.100.
Keepalived keeps the address 10.10.0.100 on the active node (MASTER).
When we run kubeadm init --control-plane-endpoint “10.10.0.100:6443”, the request goes to the VIP -> hits HAProxy -> is redirected to the first available Control Plane node.
If the first balancer goes down, the second (BACKUP) will instantly take over the IP 10.10.0.100, and our Kubernetes cluster will continue to work without any connection interruptions.

To deploy a fault-tolerant balancer, we need to put together settings from:

snipets/cloud-init-config.yaml — time zone settings and network settings
snipets/cloud-init-user.yaml — we will use the k8sadmin user, whose containerd group we will replace with haproxy
snipets/cloud-init-lb.yaml — settings specific to deploying and launching balancer nodes

Let’s create snipets/cloud-init-lb.yaml

cat << 'EOF' > snipets/cloud-init-lb.yaml

package_update: true
package_upgrade: true

packages:
  - haproxy
  - keepalived

write_files:
  - path: /etc/sysctl.d/99-haproxy.conf
    content: |
      net.ipv4.ip_nonlocal_bind = 1

  - path: /etc/haproxy/haproxy.cfg
    content: |
      global
          log /dev/log local0
          user haproxy
          group haproxy
          daemon
          stats socket /run/haproxy/admin.sock mode 660 level admin

      defaults
          log     global
          mode    tcp
          option  tcplog
          timeout connect 5000
          timeout client  50000
          timeout server  50000

      frontend k8s-api
          bind *:6443
          default_backend k8s-api-backend

      backend k8s-api-backend
          balance roundrobin
          option tcp-check
          timeout server 2h
          timeout client 2h
          # Here we specify the parameters of control plane node that we know
          server cp-1 10.10.0.21:6443 check check-ssl verify none fall 3 rise 2
          server cp-2 10.10.0.22:6443 check check-ssl verify none fall 3 rise 2
          server cp-3 10.10.0.23:6443 check check-ssl verify none fall 3 rise 2

      listen stats
          bind *:8404
          mode http
          stats enable
          stats uri /stats

  - path: /etc/keepalived/keepalived.conf
    content: |
      vrrp_script check_haproxy {
          script "killall -0 haproxy"
          interval 2
          weight 2
      }
      vrrp_instance VI_1 {
          state ${LB_STATE} # lb1 will be MASTER, lb2 will be BACKUP
          interface enp0s2
          virtual_router_id 51
          priority ${LB_PRIORITY} # for lb1 it will be 101, for lb2 it will be 100
          advert_int 1
          authentication {
              auth_type PASS
              auth_pass k8s_pwd # Replace k8s_pwd with a strong password
          }
          virtual_ipaddress {
              ${LB_IP} # general balancer address 10.10.0.100/24
          }
          track_script {
              check_haproxy
          }
      }

runcmd:
  - sysctl --system
  - systemctl enable --now haproxy
  - systemctl enable --now keepalived
EOF

Deploying HAProxy+Keepalived

Let’s create virtual machines for HAProxy+Keepalived:

To create a fault-tolerant balancer, we need two almost identical commands to run. The main difference between them is in the Keepalived settings (state and priority) and the individual IP addresses of the nodes.

Let’s start the first balancer node

# Real IP address for the interface (netplan)
export VM_IP="10.10.0.101/24"

# Parameters for keepalived.conf
export LB_IP="10.10.0.100/24"
export LB_STATE="MASTER"
export LB_PRIORITY="101"

multipass launch --name lb1 \
  --cpus 1 --memory 1G --disk 5G \
  --network name=en0,mode=manual \
  --cloud-init <(yq eval-all '
    # 1. Merging all files into a single object
    . as $item ireduce ({}; . *+ $item) |

    # 2. Running netplan apply after sysctl
    .runcmd |= (
      filter(. != "netplan apply") |
      (to_entries | .[] | select(.value == "sysctl --system") | .key) as $idx |
      .[:$idx+1] + ["netplan apply"] + .[$idx+1:]
    ) |
    .runcmd[].headComment = "" |

    # 3. Replacing a user group
    with(.users[] | select(.name == "k8sadmin");
      .groups |= sub("containerd", "haproxy")
    ) |

    #4. Configuring IP for application via Netplan
    with(.write_files[] | select(.path == "/etc/netplan/60-static-ip.yaml");
      .content |= (from_yaml | .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] | to_yaml)
    ) |

    # 5. Replace variables ${LB_...} in all write_files files
    with(.write_files[];
      .content |= sub("\${LB_STATE}", strenv(LB_STATE)) |
      .content |= sub("\${LB_PRIORITY}", strenv(LB_PRIORITY)) |
      .content |= sub("\${LB_IP}", strenv(LB_IP))
    )
  ' \
  snipets/cloud-init-config.yaml \
  snipets/cloud-init-user.yaml \
  snipets/cloud-init-lb.yaml)

And the second node of the balancer

# Real IP address for the interface (netplan)
export VM_IP="10.10.0.102/24"

# Parameters for keepalived.conf
export LB_IP="10.10.0.100/24"
export LB_STATE="BACKUP"
export LB_PRIORITY="100"

multipass launch --name lb1 \
  --cpus 1 --memory 1G --disk 5G \
  --network name=en0,mode=manual \
  --cloud-init <(yq eval-all '
    # 1. Merging all files into a single object
    . as $item ireduce ({}; . *+ $item) |

    # 2. Running netplan apply after sysctl
    .runcmd |= (
      filter(. != "netplan apply") |
      (to_entries | .[] | select(.value == "sysctl --system") | .key) as $idx |
      .[:$idx+1] + ["netplan apply"] + .[$idx+1:]
    ) |
    .runcmd[].headComment = "" |

    # 3. Replacing a user group
    with(.users[] | select(.name == "k8sadmin");
      .groups |= sub("containerd", "haproxy")
    ) |

    #4. Configuring IP for application via Netplan
    with(.write_files[] | select(.path == "/etc/netplan/60-static-ip.yaml");
      .content |= (from_yaml | .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] | to_yaml)
    ) |

    # 5. Replace variables ${LB_...} in all write_files files
    with(.write_files[];
      .content |= sub("\${LB_STATE}", strenv(LB_STATE)) |
      .content |= sub("\${LB_PRIORITY}", strenv(LB_PRIORITY)) |
      .content |= sub("\${LB_IP}", strenv(LB_IP))
    )
  ' \
  snipets/cloud-init-config.yaml \
  snipets/cloud-init-user.yaml \
  snipets/cloud-init-lb.yaml)

After launching, let’s check for the presence of IP 10.10.0.100.

Let’s go to any machine and check if the address 10.10.0.100 has appeared:

multipass exec lb1 -- ip addr show enp0s2

What to do after launch?

Check statistics: Open http://10.10.0.100:8404/stats in your browser. You will see that the backends (our control plane nodes) are marked in red (because they have not been initialized yet) — this is normal.
Launch Kubernetes: Now we can run kubeadm init on the first Control Plane node. Since VIP 10.10.0.100 is already active and HAProxy is listening on port 6443, there will be no timeout error.

Deploying the Control Plane

Let’s gather the cloud-init settings for deploying our control plane nodes. They will be similar to the ones we used to create etcd nodes, but with some differences.

According to the recommendations, we will allocate at least 2 CPU cores and 2 GB of RAM to the control panel node. For the first control panel node, we will use the IP address 10.10.0.21. We will also install HAProxy as a local load balancer for accessing etcd nodes.

Configuring a local load balancer for access to etcd nodes

To access etcd nodes, we will deploy a local load balancer. Each API server will refer to its own load balancer (127.0.0.1:2379). This approach is called “Sidecar Load Balancing” (or local proxy). It provides maximum fault tolerance: even if the network between nodes starts to “storm,” each API server will have its own local path to etcd.

Since we are doing this for a Kubernetes cluster, the best way to implement this is to use Static Pods. The node manager (kubelet) will start and maintain HAProxy itself.

Preparing the HAProxy configuration

For the control panel nodes, create a file with HAProxy settings /etc/haproxy-lbaas/haproxy.cfg to balance traffic to etcd nodes.

cat << EOF > snipets/cloud-init-cp-haproxy.yaml
write_files:
  # Configuring HAProxy to access etcd nodes
  - path: /etc/haproxy-lbaas/haproxy.cfg
    content: |
      global
          log /dev/log local0
          user haproxy
          group haproxy

      defaults
          log global
          mode tcp
          option tcplog
          timeout connect 5000ms
          timeout client 50000ms
          timeout server 50000ms

      frontend etcd-local
          bind 127.0.0.1:2379
          description "Local proxy for etcd cluster"
          default_backend etcd-cluster

      backend etcd-cluster
          option tcp-check
          # Important: we use roundrobin for load balancing
          balance roundrobin
          server etcd-1 10.10.0.11:2379 check inter 2000 rise 2 fall 3
          server etcd-2 10.10.0.12:2379 check inter 2000 rise 2 fall 3
          server etcd-3 10.10.0.13:2379 check inter 2000 rise 2 fall 3
EOF

Creating a Static Pod for HAProxy

Let’s make kubelet run HAProxy. Let’s create a manifest in the static pods folder (by default, this is /etc/kubernetes/manifests/).

Let’s create the file /etc/kubernetes/manifests/etcd-proxy.yaml with the HAProxy static pod manifest:

cat << EOF > snipets/cloud-init-cp-haproxy-manifest.yaml
write_files:
  # Manifest of static HAProxy proxy for balancing traffic to etcd nodes
  - path: /etc/kubernetes/manifests/etcd-haproxy.yaml
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: etcd-haproxy
        namespace: kube-system
        labels:
          component: etcd-haproxy
          tier: control-plane
      spec:
        containers:
        - name: etcd-haproxy
          image: haproxy:2.8-alpine # Використовуємо легкий образ
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
          - name: haproxy-config
            mountPath: /usr/local/etc/haproxy/haproxy.cfg
            readOnly: true
        hostNetwork: true # Важливо: под має бачити 127.0.0.1 хоста
        volumes:
        - name: haproxy-config
          hostPath:
            path: /etc/haproxy-lbaas/haproxy.cfg
            type: File
EOF

Deploying the first control panel node

Let’s deploy the first control panel node cp-1 with IP 10.10.0.21/24

export VM_IP="10.10.0.21/24"

multipass launch --name cp-1 \
  --cpus 2 --memory 2.5G --disk 8G \
  --network name=en0,mode=manual \
  --cloud-init <( yq eval-all '
      # Merge all files into a single object
      . as $item ireduce ({}; . *+ $item) |

      # Update network configuration
      with(.write_files[] | select(.path == "/etc/netplan/60-static-ip.yaml");
        .content |= (
          from_yaml |
          .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] |
          to_yaml
        )
      ) ' \
      snipets/cloud-init-config.yaml \
      snipets/cloud-init-user.yaml \
      snipets/cloud-init-base.yaml \
      snipets/cloud-init-cp-haproxy.yaml \
      snipets/cloud-init-cp-haproxy-manifest.yaml)

After spinning up the control panel node, copy the following files from any etcd node to the first node of the control panel (this will not be necessary for other control panel nodes during the first two hours after initialization until the Secret with keys is removed by the system).

# 1. Prepare the files on the source node (10.10.0.11):
# Copy them to a temporary folder and change the owner to the current user so that scp can read them.
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.11 " \
  mkdir -p /tmp/cert/ && \
  sudo cp /etc/kubernetes/pki/etcd/ca.crt /tmp/cert/ && \
  sudo cp /etc/kubernetes/pki/apiserver-etcd-client.* /tmp/cert/ && \
  sudo chown k8sadmin:k8sadmin /tmp/cert/* "

# 2. Transferring files between nodes via your local terminal:
# Use quotation marks to handle wildcards (*) on the remote side
scp -i ~/.ssh/k8s_cluster_key -r 'k8sadmin@10.10.0.11:/tmp/cert/' 'k8sadmin@10.10.0.21:/tmp'

# 3. Placing files on the target node (10.10.0.21):
# Create a folder (if it does not exist), move the files, and restore root privileges.
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.21 " \
  sudo mkdir -p /etc/kubernetes/pki/etcd/ && \
  sudo mv /tmp/cert/ca.crt /etc/kubernetes/pki/etcd/ && \
  sudo chown root:root /etc/kubernetes/pki/etcd/ca.crt && \
  sudo mv /tmp/cert/apiserver-etcd-client.* /etc/kubernetes/pki/ && \
  sudo chown root:root /etc/kubernetes/pki/apiserver-etcd-client.*"

#4. Cleaning temporary files:
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.11 "rm -rf /tmp/cert"
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.21 "rm -rf /tmp/cert"

Checking the use of Static Pods HAProxy

We placed the etcd-proxy.yaml manifest in /etc/kubernetes/manifests/. However, kubelet ignores this folder until it receives a command to start (this will happen after the control panel node is initialized).

In addition, during the preflight phase, the kubeadm command attempts to verify the availability of etcd before any cluster components start running. Since our HAProxy is supposed to run as a Pod, it is not yet running, port 127.0.0.1:2379 is closed, and we will get a connection refused error when attempting to initialize the control plane node.

[preflight] Running pre-flight checks
	[WARNING ExternalEtcdVersion]: Get "https://127.0.0.1:2379/version": dial tcp 127.0.0.1:2379: connect: connection refused

Checking via cURL (the fastest way)

Since we are using TLS, we will need the certificates that we have already prepared for kubeadm. Let’s try to access etcd via the local port:
```
sudo curl --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
     --key /etc/kubernetes/pki/apiserver-etcd-client.key \
     https://127.0.0.1:2379/health
```
Expected result: {“health”:“true”}. If we get this response, it means that HAProxy is successfully forwarding traffic to one of the nodes in our etcd cluster.
Checking the status of Static Pod

Let’s check if the HAProxy container has started at all. Since kubectl may not work if etcd is unavailable, use the container runtime tool (in our case, crictl):
```
# For containerd (standard for modern K8s)
sudo crictl ps | grep etcd-haproxy

# View proxy logs
sudo crictl logs $(sudo crictl ps -q --name etcd-haproxy)
```
The HAProxy logs should contain entries about successful health checks (Health check passed) for backend nodes 10.10.0.11, .12, .13.
Checking via system sockets

Let’s make sure that HAProxy is actually listening on port 2379 on the local interface:
```
sudo ss -tulpn | grep 2379
```
We should see that the process (haproxy) is listening on 127.0.0.1:2379.

Configuring kubeadm

Let’s create a file named kubeadm-config.yaml in the home directory of the k8sadmin user on the first node to initialize the control panel.

ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.21 "cat << 'EOF' > \$HOME/kubeadm-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: \"v1.34.3\"
controlPlaneEndpoint: \"10.10.0.100:6443\"
etcd:
  external:
    endpoints:
      - https://127.0.0.1:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
  serviceSubnet: \"10.96.0.0/16\"
  podSubnet: \"10.244.0.0/16\"
  dnsDomain: \"cluster.local\"
EOF"

Starting initialization and bypassing ExternalEtcdVersion verification

When kubeadm init starts, it attempts to verify access to the external etcd cluster. However, we have “wrapped” access to the cluster in a local HAProxy, which kubelet runs as a static pod. However, at this point, we have not yet started kube-apiserver, which will take over the management of kubelet, which in turn will start the pod with haproxy. Therefore, we need to disable “preflight checks” for etcd (--ignore-preflight-errors=ExternalEtcdVersion). To initialize the cluster on the control plane node, run the following command:

sudo kubeadm init \
  --config $HOME/kubeadm-config.yaml \
  --upload-certs \
  --ignore-preflight-errors=ExternalEtcdVersion

What to look for during initialization:

After running the command, watch for the [control-plane] Creating static pod manifest for “kube-apiserver” step. If the API server starts successfully, it means that it was able to connect to etcd through your local proxy.

If the command stops again with an error, check if there are any processes left in the system from previous attempts:

# If you need to completely reset the status before a new attempt
sudo kubeadm reset -f
# After the reset, you will need to restart kubelet again to bring up the proxy.
sudo systemctl restart kubelet

Under normal circumstances, you will see the following log of kubeadm init running

View log

[init] Using Kubernetes version: v1.34.3
[preflight] Running pre-flight checks
	[WARNING ExternalEtcdVersion]: Get "https://127.0.0.1:2379/version": dial tcp 127.0.0.1:2379: connect: connection refused
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cp-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.176 10.10.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.932756ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.2.176:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 1.515332016s
[control-plane-check] kube-scheduler is healthy after 19.381430245s
[control-plane-check] kube-apiserver is healthy after 21.504025006s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
7a088e936453ab3143f25cdb9827b8cac60888c75f91b9d6c2d08d23a32a2bc9
[mark-control-plane] Marking the node cp-1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node cp-1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: z28v5d.4vm6rzekoibear23
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes running the following command on each as root:

  kubeadm join 10.10.0.100:6443 --token z28v5d.4vm6rzekoibear23 \
	--discovery-token-ca-cert-hash sha256:4c23033729b477d1fc30ae4b4041fe7dae70fa8defd5ecb57c571e969e00f8e0 \
	--control-plane --certificate-key 7a088e936453ab3143f25cdb9827b8cac60888c75f91b9d6c2d08d23a32a2bc9

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.0.100:6443 --token z28v5d.4vm6rzekoibear23 \
	--discovery-token-ca-cert-hash sha256:4c23033729b477d1fc30ae4b4041fe7dae70fa8defd5ecb57c571e969e00f8e0

After starting initialization with error ignoring, it is important to ensure that the API server was able to connect to the database and is not simply “hanging” in a waiting state.

Main check: API server status

If kubeadm init has passed the preflight stage, it will attempt to start kube-apiserver. If the API server cannot communicate with etcd through our proxy, it will continuously restart.

Let’s check the API server logs:

sudo tail -f /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log

View API server log

k8sadmin@cp-1:~$ sudo tail -f /var/log/pods/kube-system_kube-apiserver-cp-1_70e58895431aff7a0cb441009519f1c6/kube-apiserver/0.log
2026-01-07T11:10:20.602046303+02:00 stderr F W0107 09:10:20.601902       1 logging.go:55] [core] [Channel #359 SubChannel #360]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
2026-01-07T11:10:20.617773217+02:00 stderr F W0107 09:10:20.617570       1 logging.go:55] [core] [Channel #363 SubChannel #364]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
2026-01-07T11:10:20.634721419+02:00 stderr F W0107 09:10:20.634607       1 logging.go:55] [core] [Channel #367 SubChannel #368]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
2026-01-07T11:10:20.644114716+02:00 stderr F W0107 09:10:20.644002       1 logging.go:55] [core] [Channel #371 SubChannel #372]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
2026-01-07T11:10:20.662781139+02:00 stderr F W0107 09:10:20.662426       1 logging.go:55] [core] [Channel #375 SubChannel #376]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
2026-01-07T11:10:20.675026234+02:00 stderr F W0107 09:10:20.674872       1 logging.go:55] [core] [Channel #379 SubChannel #380]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: authentication handshake failed: context canceled"
2026-01-07T11:10:20.860920026+02:00 stderr F W0107 09:10:20.860664       1 logging.go:55] [core] [Channel #383 SubChannel #384]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"
2026-01-07T11:11:10.4594371+02:00 stderr F I0107 09:11:10.459184       1 controller.go:667] quota admission added evaluator for: replicasets.apps
2026-01-07T11:19:59.613078541+02:00 stderr F I0107 09:19:59.612638       1 cidrallocator.go:277] updated ClusterIP allocator for Service CIDR 10.96.0.0/16

This log indicates a very important stage: our kube-apiserver has successfully started, but the initialization process went through a “struggle” to connect to etcd.

Here’s what happened:

Error stage (Handshake failed)

The first lines of the log show errors: transport: authentication handshake failed: context canceled and dial tcp 127.0.0.1:2379: operation was canceled.

This means that:
- The API server tried to connect to your HAProxy (127.0.0.1:2379).
- The connection was established, but the TLS handshake was interrupted.
Reason: This happened at the same moment when kubeadm was still generating or substituting certificates, or when HAProxy had not yet established a stable session with the etcd backend nodes. This is normal behavior during a “cold” start of the control panel.
Success phase (Stabilization)

Note the last lines: I0107 09:11:10.459184 ... quota admission added evaluator for: replicasets.apps, I0107 09:19:59.612638 ... updated ClusterIP allocator for Service CIDR 10.96.0.0/16

This is a win:
- The API server is alive. If it couldn’t connect to etcd, it would just crash (CrashLoopBackOff) and you wouldn’t see any logs about cidrallocator.
The ClusterIP allocator message means that the API server has already started writing data to etcd and managing cluster resources.

The interval between entries (10 minutes) shows stable background operation of the controllers.

Component status

The fact that you see folders for kube-controller-manager and kube-scheduler in /var/log/pods/ confirms that kubeadm has successfully completed the manifest creation phase and all three core Control Plane components are running.

k8sadmin@cp-1:~$ sudo ls -la /var/log/pods/
total 28
drwxr-x---  7 root root   4096 Jan  8 22:23 .
drwxr-xr-x 11 root syslog 4096 Jan  8 22:23 ..
drwxr-xr-x  3 root root   4096 Jan  8 22:23 kube-system_etcd-haproxy-cp-1_e2b3a81fe56706e845a17ba096c5dfad
drwxr-xr-x  3 root root   4096 Jan  8 22:23 kube-system_kube-apiserver-cp-1_e74afa6943effdf6bbdcfc384bd87bb6
drwxr-xr-x  3 root root   4096 Jan  8 22:23 kube-system_kube-controller-manager-cp-1_b635ba5e5439cc2c581bf61ca1e6fb9e
drwxr-xr-x  3 root root   4096 Jan  8 22:23 kube-system_kube-proxy-9qh54_9e21026d-0d6e-4f8c-a071-842149ffd24e
drwxr-xr-x  3 root root   4096 Jan  8 22:23 kube-system_kube-scheduler-cp-1_0cf013b3f4c49c84241ee3a56735a15d

What to check now?

Since the API server is responding, run the following commands to finally verify that the first node is working:

Checking nodes: kubectl get nodes (You should see cp-1 in NotReady status — this is normal because we haven’t installed Calico yet).
Proxy health check (via HAProxy): kubectl get --raw /healthz/etcd (Should return ok).
Check etcd access points: kubectl describe pod kube-apiserver-cp-1 -n kube-system | grep etcd (Make sure that only https://127.0.0.1:2379 appears there).
Check HAProxy: follow the link http://10.10.0.100:8404/stats to the load balancer dashboard in the control panel (In the k8s-api-backend section, the status of the first control panel node should be UP)

Next, it is recommended to deploy CNI plugins for the event network.

Installing Calico

We will use Calico to create a pods network. The article “Install Calico networking and network policy for on-premises deployments” describes in detail the process of installing Calico on your own equipment.

We will use Tigera Operator and custom resource definitions (CRDs). We will apply the following two manifests:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/operator-crds.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/tigera-operator.yaml

Then we will download the file with the custom resources needed to configure Calico.

curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/custom-resources-bpf.yaml

and specify the CIDR of the pod network as we specified it when initializing the first control panel node — 10.244.0.0/16 (the default value in Calico is cidr: 192.168.0.0/16).

After making the changes, apply the manifest to install Calico

kubectl create -f custom-resources-bpf.yaml

Track the installation using the command watch kubectl get tigerastatus. After a few minutes (6-7 minutes), all Calico components will have a value of True in the AVAILABLE column.

NAME                            AVAILABLE   PROGRESSING   DEGRADED   SINCE
apiserver                       True        False         False      4m9s
calico                          True        False         False      3m29s
goldmane                        True        False         False      3m39s
ippools                         True        False         False      6m4s
kubeproxy-monitor               True        False         False      6m15s
whisker                         True        False         False      3m19s

Once the Calico components are available, the control plane node will transition to the READY state.

What to do first: deploy Calico or run kubeadm join?

Technically, you can do either, but option #1 (CNI before Join) is best practice.

Option 1: CNI first, then Join (Recommended)

When you install CNI immediately after initializing the first node (cp-1), the cluster network becomes operational immediately.

Node Status: The first node quickly transitions to Ready status.
CoreDNS: CoreDNS pods, which typically hang in Pending status without a network, start up.
Joining new nodes: When you join cp-2 and cp-3, they immediately receive network settings. System pods on new nodes will be able to start communicating with each other faster.
Convenience: You can see the actual health status of each new node immediately after it joins.

Option 2: Join first, then CNI

This is also a working scenario, but it looks more “alarming” during the process.

Node status: All nodes (cp-1, cp-2, cp-3) will be in NotReady state.
CoreDNS: All system network components will be waiting.
Joining: kubeadm join will be successful because CNI is not required for the joining process itself (TLS Bootstrap and config copying) — the physical network between nodes is used here.
Risks: If a problem arises with the network communication of the pods themselves during the join (for example, health checks of system components), it will be more difficult for you to understand whether this is a join problem or simply the absence of CNI.

We recommend following this order:

kubeadm init on cp-1.
Required kubeconfig settings.
CNI installation (e.g., Cilium, Calico, or Flannel).
Check kubectl get nodes (should be Ready).
Transfer certificates to new nodes (if necessary).
kubeadm join for cp-2 and cp-3.

Deploying and joining the subsequent control panel nodes

Let’s use the ready-made command for the first control panel node. Replace the IP address with the appropriate one for each node:

cp-2.yaml — 10.10.0.22
cp-3.yaml — 10.10.0.23

Let’s create virtual machines:

export VM_IP="10.10.0.22/24"

multipass launch --name cp-2 \
  --cpus 2 --memory 2.5G --disk 8G \
  --network name=en0,mode=manual \
  --cloud-init <( yq eval-all '
      # Merge all files into a single object
      . as $item ireduce ({}; . *+ $item) |

      # Update network configuration
      with(.write_files[] | select(.path == "/etc/netplan/60-static-ip.yaml");
        .content |= (
          from_yaml |
          .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] |
          to_yaml
        )
      ) ' \
      snipets/cloud-init-config.yaml \
      snipets/cloud-init-user.yaml \
      snipets/cloud-init-base.yaml \
      snipets/cloud-init-cp-haproxy.yaml \
      snipets/cloud-init-cp-haproxy-manifest.yaml)

Once the node is ready, we will join it as another control panel node. Using the output obtained during the initialization of the first control panel node, we will add the --ignore-preflight-errors=ExternalEtcdVersion parameter to it, just as we did during the initialization of the first control panel node:

sudo kubeadm join 10.10.0.100:6443 --token z28v5d.4vm6rzekoibear23 \
        --discovery-token-ca-cert-hash sha256:4c23033729b477d1fc30ae4b4041fe7dae70fa8defd5ecb57c571e969e00f8e0 \
        --control-plane --certificate-key 7a088e936453ab3143f25cdb9827b8cac60888c75f91b9d6c2d08d23a32a2bc9 \
        --ignore-preflight-errors=ExternalEtcdVersion

The log of joining the second control panel node will look like this

View the log of joining the second node

k8sadmin@cp-2:~$ sudo kubeadm join 10.10.0.100:6443 --token z28v5d.4vm6rzekoibear23 \
        --discovery-token-ca-cert-hash sha256:4c23033729b477d1fc30ae4b4041fe7dae70fa8defd5ecb57c571e969e00f8e0 \
        --control-plane --certificate-key 7a088e936453ab3143f25cdb9827b8cac60888c75f91b9d6c2d08d23a32a2bc9 \
        --ignore-preflight-errors=ExternalEtcdVersion
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki"
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cp-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.177 10.10.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[control-plane-join] Using external etcd - no local stacked instance added
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.005951104s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
[mark-control-plane] Marking the node cp-2 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node cp-2 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.2.177:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 2.950097ms
[control-plane-check] kube-scheduler is healthy after 6.396945ms
[control-plane-check] kube-apiserver is healthy after 501.413278ms

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

If a long time has passed between creating the first control panel node and joining subsequent nodes, you may encounter this error.

[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
error: error execution phase control-plane-prepare/download-certs: error downloading certs: error downloading the secret: Secret "kubeadm-certs" was not found in the "kube-system" Namespace. This Secret might have expired. Please, run `kubeadm init phase upload-certs --upload-certs` on a control plane to generate a new one

Since we use External Etcd, the standard upload-certs procedure does not work as described in most instructions, because Kubernetes does not store keys to external databases in its secrets for security reasons.

If we need to add the next node or restore the current one, we need to perform a few additional steps.

This method is used when automatic key loading (--certificate-key) is not possible or the secret with the keys has expired.

Preparing the file system on the new node

On the new node (for example, cp-4), create the necessary folders for certificates and manifests.
```
# На новому вузлі
sudo mkdir -p /etc/kubernetes/pki/etcd
sudo mkdir -p /etc/kubernetes/manifests
```

Manual transfer of certificates

We need to copy 9 files from any working control panel node (for example, from cp-1) to the new node. The paths must match exactly.

File on source (cp-1)	Where to place on new (cp-4)	Description
`/etc/kubernetes/pki/ca.crt`	`/etc/kubernetes/pki/ca.crt`	Cluster CA
`/etc/kubernetes/pki/ca.key`	`/etc/kubernetes/pki/ca.key`	CA key
`/etc/kubernetes/pki/sa.pub`	`/etc/kubernetes/pki/sa.pub`	ServiceAccount key (public)
`/etc/kubernetes/pki/sa.key`	`/etc/kubernetes/pki/sa.key`	ServiceAccount key (private)
`/etc/kubernetes/pki/front-proxy-ca.crt`	`/etc/kubernetes/pki/front-proxy-ca.crt`	CA for API aggregation
`/etc/kubernetes/pki/front-proxy-ca.key`	`/etc/kubernetes/pki/front-proxy-ca.key`	Front Proxy Key
`/etc/kubernetes/pki/apiserver-etcd-client.crt`	`/etc/kubernetes/pki/apiserver-etcd-client.crt`	Client certificate for etcd
`/etc/kubernetes/pki/apiserver-etcd-client.key`	`/etc/kubernetes/pki/apiserver-etcd-client.key`	Client key for etcd
`/etc/kubernetes/pki/etcd/ca.crt`	`/etc/kubernetes/pki/etcd/ca.crt`	CA for etcd itself

Important: After copying, set the correct permissions for the key files (.key), otherwise kubeadm may complain about security:

sudo chmod 600 \
  /etc/kubernetes/pki/apiserver-etcd-client.key \
  /etc/kubernetes/pki/ca.key \
  /etc/kubernetes/pki/front-proxy-ca.key  \
  /etc/kubernetes/pki/sa.key

Note: If you do not want to manually copy all 9 key and certificate files, but only copy the files /etc/kubernetes/pki/etcd/ca.crt, /etc/kubernetes/pki/apiserver-etcd-client.key, and /etc/kubernetes/pki/apiserver -etcd-client.crt files to the target machine, you can first run the join command with the --certificate-key parameter (as in the case of Stacked Etcd), which will help copy most of the necessary files managed by the apiserver. After that, manually copy the three files for etcd and run the join command without the --certificate-key parameter.

To speed up the copying process, you can use the following script

cat << 'EOF' > k8s-transit.sh
#!/bin/bash

# How to use
#  “Usage: $0 <SRC_IP> <DST_IP> <USER> [SSH_KEY_PATH]”
#  “Example: k8s-transit.sh 10.10.0.21 10.10.0.178 k8sadmin ~/.ssh/id_rsa”

# --- DEFAULT VALUES ---
DEFAULT_SRC="10.10.0.21"
DEFAULT_DST="10.10.0.178"
DEFAULT_USR="k8sadmin"
DEFAULT_KEY="~/.ssh/k8s_cluster_key"
# ---------------------------

SRC=${1:-$DEFAULT_SRC}
DST=${2:-$DEFAULT_DST}
USR=${3:-$DEFAULT_USR}
KEY_PATH=${4:-$DEFAULT_KEY}

# Create an option for the SSH key
SSH_OPTS=""
if [ -f "$(eval echo $KEY_PATH)" ]; then
    SSH_OPTS="-i $KEY_PATH"
fi

echo “🚀 Using configuration:”
echo “   Source (SRC):      $SRC”
echo “   Destination (DST): $DST”
echo “   User:              $USR”
echo "   SSH key:           ${KEY_PATH:-'default'}"
echo "----------------------------------------------------"

# List of files
FILES=(
    "/etc/kubernetes/pki/ca.crt"
    "/etc/kubernetes/pki/ca.key"
    "/etc/kubernetes/pki/sa.pub"
    "/etc/kubernetes/pki/sa.key"
    "/etc/kubernetes/pki/front-proxy-ca.crt"
    "/etc/kubernetes/pki/front-proxy-ca.key"
    "/etc/kubernetes/pki/apiserver-etcd-client.crt"
    "/etc/kubernetes/pki/apiserver-etcd-client.key"
    "/etc/kubernetes/pki/etcd/ca.crt"
)

# 1. Making the folders
ssh $SSH_OPTS -t $USR@$DST "sudo mkdir -p /etc/kubernetes/pki/etcd"

# 2. File transfer
for FILE in "${FILES[@]}"; do
    echo "Transfer: $FILE"
    ssh $SSH_OPTS $USR@$SRC "sudo cat $FILE" | ssh $SSH_OPTS $USR@$DST "sudo tee $FILE > /dev/null"
done

# 3. Owner and access rights settings
echo "🔒 Change owner to root and set permissions to $DST..."
ssh $SSH_OPTS -t $USR@$DST " \
    sudo chown -R root:root /etc/kubernetes/pki && \
    sudo chmod 644 /etc/kubernetes/pki/*.crt /etc/kubernetes/pki/sa.pub && \
    sudo chmod 600 /etc/kubernetes/pki/*.key /etc/kubernetes/pki/sa.key"

echo "✅ Successfully completed!"
EOF

chmod +x k8s-transit.sh

Configuring access to etcd (Local Proxy)

Since we are using an architecture where each node communicates with etcd through a local proxy (HAProxy), let’s make sure that the proxy manifest is already on the new node.

If not, copy the etcd-haproxy.yaml file from cp-1 to the /etc/kubernetes/manifests/ directory on the new node. This ensures that as soon as kubelet starts, it will establish a connection to the database.
Generating the Join command (on the running node)

On the running node (cp-1), generate a token and hash. We don’t need a certificate key because we already transferred the files manually.
```
# On cp-1
kubeadm token create --print-join-command
```
As a result, we will get a command that looks like this: kubeadm join 10.10.0.100:6443 --token <token> --discovery-token-ca-cert-hash <hash>

Start the join (on the new node)

Run the command on the new node, adding the --control-plane flag. ⚠️ Do not add --certificate-key.

# On the new node
sudo kubeadm join 10.10.0.100:6443 \
   --token <your_token> \
   --discovery-token-ca-cert-hash sha256:<your_hash> \
   --control-plane \
   --ignore-preflight-errors=ExternalEtcdVersion

View node joining log

k8sadmin@cp-4:~$ sudo kubeadm join 10.10.0.100:6443 --token efwfq8.puh8nqqk3a3dqqdx --discovery-token-ca-cert-hash sha256:4c23033729b477d1fc30ae4b4041fe7dae70fa8defd5ecb57c571e969e00f8e0 --control-plane --ignore-preflight-errors=ExternalEtcdVersion
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cp-4 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.179 10.10.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[control-plane-join] Using external etcd - no local stacked instance added
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 503.676345ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
[mark-control-plane] Marking the node cp-4 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node cp-4 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.2.179:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-scheduler is healthy after 9.840373ms
[control-plane-check] kube-controller-manager is healthy after 11.326085ms
[control-plane-check] kube-apiserver is healthy after 21.681901ms

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

If you have Stacked Etcd, the following command will be sufficient to obtain new instructions for joining nodes to the cluster. In this architecture, kubeadm fully automates the process: it downloads all the necessary certificates (including those required to create a new etcd member node) into the kubeadm-certs secret.

# On node cp-1
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.21
sudo -sE
$(kubeadm token create --print-join-command) --control-plane --certificate-key $(kubeadm init phase upload-certs --upload-certs | tail -1)

This works for Stacked Etcd:

upload-certs: Unlike External Etcd, in Stacked mode kubeadm knows about all local etcd certificates and packs them into a bundle.
--certificate-key: You get a decryption key that allows the new node to automatically download this package to its /etc/kubernetes/pki/ directory.
Automation: The new node does not need to copy anything manually — it will create both Kubernetes certificates and a local copy of etcd, joining it to the existing participants.

This command works great, but: kubeadm init phase upload-certs --upload-certs uploads certificates to the secret for only 2 hours.

If you plan to add nodes later, you will have to run this command again. If you are doing this “here and now,” this is the ideal and fastest way.

Tip: If there have already been failed attempts to join on the new node, be sure to run sudo kubeadm reset -f before running the new command.

Disconnecting and replacing a control panel node

Removing a control panel node if it fails or to replace it has a significant advantage in an architecture with External Etcd: since the database is outside the Kubernetes nodes, you only risk API availability, not data integrity.

To remove a control plane node (for example, cp-2) from the cluster, follow these steps.

Prepare the cluster

Perform these steps on another healthy node (for example, cp-1) or communicate with the control panel via the HAProxy balancer address.
- Remove the load from the node: Allow current tasks to complete and prevent new pods from being assigned
```
kubectl cordon cp-2
kubectl drain cp-2 --ignore-daemonsets --delete-emptydir-data
```
- Remove the node from the Kubernetes registry: This will delete the Node object and associated certificates (if auto-rotation is used).
```
kubectl delete node cp-2
```
Cleaning up the node itself (on cp-2)

Now go to the node you want to disconnect.
- Resetting kubeadm settings: This will remove static pod manifests (kube-apiserver, controller-manager, etc.) and configuration files in /etc/kubernetes.
```
sudo kubeadm reset -f
```
- Clearing IPVS/Iptables:
```
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

# If you used IPVS:
sudo ipvsadm --clear
```
- Removing residual files: It is recommended to delete the certificate folder to avoid conflicts when reconnecting.
```
sudo rm -rf /etc/kubernetes/pki
```
Cleaning up Etcd

This is where the main difference between the architectures comes into play.

For External Etcd:

Since etcd runs separately, nothing needs to be done. The external etcd cluster doesn’t even know that the API server on cp-2 has stopped working. You have simply removed one of the database clients.

For Stacked Etcd:

This is the most important step. Since etcd was running locally on cp-2, it was part of the quorum. If you simply shut down the node, the etcd cluster will consider it “down” and wait for it to return, which can negatively affect the quorum.
1. You need to log in to another master node.
2. Find the etcd member ID for cp-2:
```
kubectl exec -n kube-system etcd-cp-1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list
```
3. Remove this ID:
```
kubectl exec -n kube-system etcd-cp-1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member remove <MEMBER_ID>
```
Load Balancer Update

If you have an external balancer in front of your control panel (e.g., HAProxy or cloud LB), be sure to remove the cp-2 IP address from the backend configuration.

If you are using local HAProxy on each node (as we configured earlier), the other nodes will simply stop referring to cp-2, but the configuration on the worker nodes should be updated so that they do not waste time trying to connect to the missing control panel node.
Replacing a node (if necessary)

If you want to replace cp-2 with a new node with the same name:

Prepare a new OS.
Follow the steps from the previous instructions (copy 9 certificate files + proxy manifest).
Run kubeadm join.

Deploying Worker Nodes

To deploy worker nodes, we will use the settings from snipets/cloud-init-config.yaml, snipets/cloud-init-user.yaml, and snipets/cloud-init-base.yaml. During the file merging process, we will remove kubectl from the list of packages to install. For worker nodes, we will use IP addresses from the range 10.10.0.30 — 10.10.0.99.

export VM_IP="10.10.0.31/24"

multipass launch --name wn-1 \
  --cpus 2 --memory 2G --disk 10G \
  --network name=en0,mode=manual \
  --cloud-init <( yq eval-all '
      # Merge all files into a single object
      . as $item ireduce ({}; . *+ $item) |

      # Removing kubectl from the package list
      del(.packages[] | select(. == "kubectl")) |

      # Update network configuration
      with(.write_files[] | select(.path == "/etc/netplan/60-static-ip.yaml");
        .content |= (
          from_yaml |
          .network.ethernets.enp0s2.addresses += [strenv(VM_IP)] |
          to_yaml
        )
      ) ' \
      snipets/cloud-init-config.yaml \
      snipets/cloud-init-user.yaml \
      snipets/cloud-init-base.yaml )

Once the virtual machine has been provisioned, we will log into it via ssh.

ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.31

And let’s join it to the cluster using the command we received during the initialization of the control panel:

sudo kubeadm join 10.10.0.100:6443 --token jfugsz.51c9pp44fwifcu94 \
  --discovery-token-ca-cert-hash sha256:736ca5cf47fca3f4c1fd9c414f6ba70e4fefdb2f52deec5e2526f5ebccf838d6

The Calico operator (tigera-operator) will register the node in the event network, and in a minute or two, the status of the working node will change to READY. The node is ready to receive a workload.

Installing MetalLB

For instructions on installing MetalLB to balance access to worker nodes, refer to the article “Deploying a Kubernetes cluster on a local computer: A Complete Step-by-Step Guide”.

Your cluster is now ready to accept workloads.

Testing the cluster

multipass exec cp1 -- kubectl get nodes
multipass exec cp1 -- kubectl get pods -A

Summary

This guide demonstrates a complete deployment of an HA Kubernetes cluster with an external etcd topology. The cluster is ready to use and can be easily scaled. All steps follow the DRY principle and can be adapted to different environments.

Appendices

Network architecture for HA Kubernetes with external etcd

Look…

Recommended network topology — Subnet: 10.10.0.0/22 (1024 addresses)

This subnet is large enough for expansion and is logically divided into blocks:

┌─────────────────────────────────────────────────────────┐
│ 10.10.0.0/22      - Kubernetes home subnet.             │
├─────────────────────────────────────────────────────────┤
│ 10.10.0.0/26      - Infrastructure (64 addresses)       │
│   10.10.0.1       - Gateway (macOS host)                │
│   10.10.0.2       - DNS (optional)                      │
│   10.10.0.10      - HAProxy/Load Balancer               │
│   10.10.0.11-20   - Infrastructure reserve.             │
├─────────────────────────────────────────────────────────┤
│ 10.10.0.64/26     - etcd cluster (64 addresses)         │
│   10.10.0.65      - etcd-1                              │
│   10.10.0.66      - etcd-2                              │
│   10.10.0.67      - etcd-3                              │
│   10.10.0.68-70   - Reserve for additional etcd         │
├─────────────────────────────────────────────────────────┤
│ 10.10.0.128/25    - Control Plane (128 addresses)       │
│   10.10.0.129     - k8s-master-1                        │
│   10.10.0.130     - k8s-master-2                        │
│   10.10.0.131     - k8s-master-3                        │
│   10.10.0.132-140 - Reserve for additional masters      │
├─────────────────────────────────────────────────────────┤
│ 10.10.1.0/24      - Worker Nodes (256 addresses)        │
│   10.10.1.10      - k8s-worker-1                        │
│   10.10.1.11      - k8s-worker-2                        │
│   10.10.1.12      - k8s-worker-3                        │
│   10.10.1.13      - k8s-worker-4                        │
│   10.10.1.14      - k8s-worker-5                        │
│   10.10.1.15-100  - Reserve for additional workers      │
├─────────────────────────────────────────────────────────┤
│ 10.10.2.0/24   - Pod Network (256 addresses) - optional │
│   Used for testing.                                     │
├─────────────────────────────────────────────────────────┤
│ 10.10.3.0/24   - Service Network - optional             │
│   Used for testing.                                     │
└─────────────────────────────────────────────────────────┘

Detailed IP address plan

1. Infrastructure (10.10.0.0/26)

IP address	Hostname	Role
10.10.0.1	macOS-host	Gateway/Bridge
10.10.0.10	haproxy-lb	Load Balancer (optional)

2. etcd Cluster (10.10.0.64/26)

IP address	Hostname	Role	vCPU	RAM	Disk
10.10.0.65	etcd-1	etcd member	2	2GB	20GB
10.10.0.66	etcd-2	etcd member	2	2GB	20GB
10.10.0.67	etcd-3	etcd member	2	2GB	20GB

Minimum etcd configuration: 3 nodes for quorum

3. Control Plane (10.10.0.128/25)

IP address	Hostname	Role	vCPU	RAM	Disk
10.10.0.129	k8s-master-1	Control Plane (Primary)	2	4GB	40GB
10.10.0.130	k8s-master-2	Control Plane	2	4GB	40GB
10.10.0.131	k8s-master-3	Control Plane	2	4GB	40GB

Minimum HA configuration: 3 master nodes

4. Worker Nodes (10.10.1.0/24)

IP address	Hostname	Role	vCPU	RAM	Disk
10.10.1.10	k8s-worker-1	Worker Node	2	4GB	50GB
10.10.1.11	k8s-worker-2	Worker Node	2	4GB	50GB
10.10.1.12	k8s-worker-3	Worker Node	2	4GB	50GB

Recommendation: Minimum 2-3 workers for testing, can scale to 100+

Kubernetes internal networks (do not conflict with VMs)

# Pod Network (CNI - Calico/Flannel/Weave)
--pod-network-cidr=192.168.0.0/16

# Service Network
--service-cidr=172.16.0.0/16

# Cluster DNS
--cluster-dns=172.16.0.10

These networks are virtual and do not conflict with VM addresses 10.10.x.x

Alternative subnet options

Option 1: Compact topology (10.10.10.0/24)

For a small cluster (up to 10 nodes):

10.10.10.0/24 - Main subnet (256 addresses)
├─ 10.10.10.1      - Gateway
├─ 10.10.10.10-12  - etcd (3 nodes)
├─ 10.10.10.20-22  - Control Plane (3 masters)
└─ 10.10.10.30-50  - Workers (up to 20 workers)

Option 2: Extended topology (10.10.0.0/16)

For a large production cluster:

10.10.0.0/16 - All subnet (65536 addresses)
├─ 10.10.1.0/24   - Infrastructure
├─ 10.10.10.0/24  - etcd cluster
├─ 10.10.20.0/24  - Control Plane
├─ 10.10.30.0/22  - Workers (1024 addresses)
└─ 10.10.40.0/22  - Reserve for expansion

Option 3: Classic private network (172.16.0.0/16)

Alternative if 10.10.x.x is already taken:

172.16.0.0/16
├─ 172.16.1.0/24   - Infrastructure + etcd
├─ 172.16.2.0/24   - Control Plane
└─ 172.16.10.0/23  - Workers (512 addresses)

DNS records (recommended)

Add to /etc/hosts on macOS:

# Kubernetes HA Cluster
10.0.1       gateway.k8s.local
10.0.10      lb.k8s.local haproxy.k8s.local

# etcd cluster
10.0.65      etcd-1.k8s.local
10.0.66      etcd-2.k8s.local
10.0.67      etcd-3.k8s.local

# Control Plane
10.0.129     master-1.k8s.local k8s-master-1
10.0.130     master-2.k8s.local k8s-master-2
10.0.131     master-3.k8s.local k8s-master-3

# Workers
10.1.10      worker-1.k8s.local k8s-worker-1
10.1.11      worker-2.k8s.local k8s-worker-2
10.1.12      worker-3.k8s.local k8s-worker-3

# API endpoint (points to LB or any master)
10.0.10      api.k8s.local kubernetes.k8s.local

Troubleshooting

Look…

Fix `--pod-network-cidr`

Q: After installing CNI plugin (Flannel), I noticed that I forgot to use --pod-network-cidr=10.244.0.0/16 when initializing an HA cluster with three control planes. How can I fix this without deleting the cluster?

Answer

To fix the CIDR of the Pod network in a running HA cluster (3 control planes) without completely reinstalling it, you need to update the kube-apiserver configuration, the kubeadm settings in ClusterConfiguration, the kube-proxy settings, and the configuration of Flannel itself.

1. Update the ClusterConfiguration

Kubernetes stores the initialization settings in the ConfigMap. These need to be changed so that future operations (such as joining new nodes) use the correct CIDR.

Увше ConfigMap:

kubectl edit configmap kubeadm-config -n kube-system

Find the networking section and change or add the podSubnet parameter:
```
networking:
  podSubnet: 10.244.0.0/16
```

2. Update Control Plane Manifests (on all 3 nodes)

On each of the three control plane nodes, you need to update the static manifest kube-controller-manager so that it correctly allocates IP addresses to the nodes.

Open the file: sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml

Find the --cluster-cidr flag and set it:

- --cluster-cidr=10.244.0.0/16
- --allocate-node-cidrs=true

After saving the file, Kubelet will automatically restart the component. Repeat this on all three master nodes.

3. Update kube-proxy settings

kube-proxy uses CIDR to configure IPTables/IPVS rules.

Edit the ConfigMap for kube-proxy:

kubectl edit configmap kube-proxy -n kube-system

Find the clusterCIDR parameter and change it:
```
clusterCIDR: "10.244.0.0/16"
```
Restart all kube-proxy Pods to apply the changes:
```
kubectl rollout restart daemonset kube-proxy -n kube-system
```
Everything should work now. If it doesn’t, skip to the next steps.

4. Reinstall Flannel with the correct CIDR

Since Flannel is already installed with incorrect (or no) settings, it needs to be reconfigured.

Download the latest Flannel manifest from the official repository:

curl -sLO https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Check the net-conf.json section in the downloaded file. It should contain:
```
"Network": "10.244.0.0/16"
```

Remove the old Flannel and install the updated one:

kubectl delete -f kube-flannel.yml # if you have the old file
kubectl apply -f kube-flannel.yml

5. Update Node Specs (PodCIDR)

This is the most critical step. Each node in the cluster already has an old (or empty) podCIDR assigned to it in its properties. These need to be updated manually.

For each node (master and worker), run:

kubectl patch node <node-name> -p '{"spec":{"podCIDR":"10.244.X.0/24", "podCIDRs":["10.244.X.0/24"]}}'

Where X is a unique number for each node (e.g. 0, 1, 2).

6. Final Reboot

For the changes to take effect and Pods to receive addresses from the new range:

Reboot the nodes one by one or delete all existing Pods in all namespaces so that they are recreated with the new IPs.
Check the network status:

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

Important: If you already have workloads running, their IP addresses will not change automatically until the Pods are recreated. On an HA cluster, this process can be done one by one for each node (cordon -> drain -> reboot -> uncordon) to avoid downtime.

Resetting etcd to its original state

Q: How do I clear the data in a cluster? Reset it to its original state?

Answer

To reset etcd to its “original” (empty) state, you need to completely delete its data on each node. Since etcd is a critical component of Kubernetes, this process will effectively destroy all of your cluster configuration (Deployment, Services, Secrets, etc.).

Step-by-step instructions for a cluster deployed via kubeadm:

1. Stop services that depend on etcd

On each master node (control-plane), you need to temporarily remove the manifests of static pods so that Kubernetes stops trying to restart etcd and the API server:

sudo mv /etc/kubernetes/manifests/*.yaml /tmp/

Wait a few seconds for the containers to stop. You can check via crictl ps or docker ps.

2. Clear data on all nodes

Delete the contents of the etcd data directory (usually /var/lib/etcd). This needs to be done on all nodes where etcd was running:

sudo rm -rf /var/lib/etcd/*

3. Initialize a new “empty” cluster

Now you need to get etcd to start as a new cluster.

Option A (Full cluster reset via kubeadm):

If you want to start from scratch (including certificates and K8s configuration), run on each node:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d
# After that, run 'kubeadm init' again on the first node

Option B (Only clean etcd data without reinstalling K8s):

If the certificates in /etc/kubernetes/pki/etcd/ are still valid, you can try to recreate only the database. Revert the manifests:

sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/

Important: If --initial-cluster-state=existing is specified in etcd.yaml, you may need to manually change it to new or use kubeadm init phase etcd local to generate a correct manifest for the new initialization.

4. Cleanup via etcdctl (if cluster is running)

If the cluster is “alive” but you just want to delete all data (keys) without stopping the services:

etcdctl \
  --endpoints https://127.0.0.1:2379 \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/peer.crt \
  --key /etc/kubernetes/pki/etcd/peer.key \
  del "" --prefix

This will delete all Kubernetes objects but will preserve the etcd cluster structure itself.

Warning: After a complete cleanup of /var/lib/etcd data, the API server will not be able to start until you reinitialize it via kubeadm init.

Kubeconfig

Q: How do I properly migrate kubeconfig to localhost and add the cluster to the list of other clusters?

Answer

To properly migrate kubeconfig and merge it with your existing clusters, it is best to avoid manually editing the main configuration file, as this often leads to YAML indentation errors.

1. Upload the file to localhost

Use ssh to read the file directly into a new local file. Replace 10.10.0.21 with the IP of your cp-1.

# On your local machine
ssh -i ~/.ssh/k8s_cluster_key k8sadmin@10.10.0.21 "sudo cat /etc/kubernetes/admin.conf" > kubeconfig-new.yaml

2. Clean up the configuration (Important)

The admin.conf file generated by kubeadm usually has standard names (kubernetes-admin, kubernetes). If you have multiple clusters, these names will conflict.

Open kubeconfig-new.yaml in an editor and rename the key fields. For example, if your cluster is called prod-cluster:

clusters.name: kubernetes ➡️ prod-cluster
contexts.name: kubernetes-admin@kubernetes ➡️ admin@prod-cluster
contexts.context.cluster: kubernetes ➡️ prod-cluster
contexts.context.user: kubernetes-admin ➡️ admin-prod
users.name: kubernetes-admin ➡️ admin-prod

Tip: If you are using a Load Balancer (10.10.0.100), make sure that the server field specifies this IP address, not the internal IP cp-1.

3. Merge with the main kubeconfig

Instead of copying text, use kubectl to merge multiple files via an environment variable.

Temporarily merge the files in memory:

export KUBECONFIG=~/.kube/config:./kubeconfig-new.yaml

Write the result to a new file (flattened):

kubectl config view --flatten > ~/.kube/config_new

Replace the old config with the new one:

mv ~/.kube/config_new ~/.kube/config
chmod 600 ~/.kube/config

4. Check and switch

Now you can see all your clusters:

# List contexts
kubectl config get-contexts

# Switch to new cluster
kubectl config use-context admin@prod-cluster

# Check connectivity
kubectl get nodes

Alternative (if you are using multiple files)

If you do not If you want to mix everything into a single ~/.kube/config file, you can simply keep them separate in ~/.kube/configs/ and add the path to them in your .bashrc or .zshrc:

export KUBECONFIG=$HOME/.kube/config:$HOME/.kube/configs/prod-cluster.yaml

Or another option.

The kubectl config command set allows you to manipulate the configuration much cleaner. Since kubeadm always creates standard names (kubernetes, kubernetes-admin), the best strategy is to import the file with the new name context and kubectl will automatically pull in the cluster and user associated with it.

Import and rename script

This approach does not edit temp-k8s.yaml, but immediately merges it into the main config with the new names.

# 1. Upload the file
ssh k8sadmin@10.10.0.21 "sudo cat /etc/kubernetes/admin.conf" > temp-k8s.yaml

# 2. Determine the name of the new cluster
NEW_NAME="k8s-cluster-prod"

# 3. Add the cluster, user, and context to your main ~/.kube/config
# We pull the data from the temporary file using 'view'

# Add the cluster
kubectl config set-cluster "$NEW_NAME" \
  --server=$(kubectl config view --kubeconfig=temp-k8s.yaml -o jsonpath='{.clusters[0].cluster.server}') \
  --certificate-authority-data=$(kubectl config view --kubeconfig=temp-k8s.yaml --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')

# Add the user
kubectl config set-credentials "admin-$NEW_NAME" \
  --client-certificate-data=$(kubectl config view --kubeconfig=temp-k8s.yaml --raw -o jsonpath='{.users[0].user.client-certificate-data}') \
  --client-key-data=$(kubectl config view --kubeconfig=temp-k8s.yaml --raw -o jsonpath='{.users[0].user.client-key-data}')

# Creating context
kubectl config set-context "$NEW_NAME" \
  --cluster="$NEW_NAME" \
  --user="admin-$NEW_NAME"

# 4. Delete the temporary file
rm temp-k8s.yaml

Adding a static IP address to Multipass virtual machines on macOS

2025-12-26T06:30:00+00:00

This guide contains a detailed description of how to add a static IP address to a Multipass virtual machine on macOS. A general description of how to do this can be found in the official documentation in the Configure static IPs section, however on macOS without certain modifications it is not possible to follow these recommendations.

Adding IP address to the first network interface of the virtual machine

Finding Multipass bridge

To access virtual machines, Multipass (on macOS) creates a bridge named bridge100. This bridge is used to provide virtual machines with access to the host network. IP addresses are assigned via DHCP, and when a virtual machine is (re)created, it is assigned the next available IP address from the bridge network.

Running the multipass networks command should show you a list of available network interfaces.

As you can see in this list, our bridge is not there. The result of multipass networks confirms the main limitation of Multipass on macOS: it “sees” only physical network adapters (Wi-Fi, Ethernet, USB). Created virtual bridges (bridge100) are ignored because they do not have the hardware profile that the Apple virtualization driver expects.

Let’s try to find the bridge another way.

# Find bridge (usually bridge100)
ifconfig | grep -A 2 "^bridge"

The response should be similar to the following:

Remember the bridge name (for example, bridge100). We will use it further.

In my case, the bridge is available at address 192.168.2.1/24 and the DHCP server allocates addresses to virtual machines from the range 192.168.2.X.

ifconfig -v bridge100

Configuring bridge on the host

Assume that we need to provide virtual machines with static IP addresses for the network 10.10.0.0/24. For this, we will create an alias to our existing bridge.

# Add IP address to bridge
sudo ifconfig bridge100 10.10.0.1/24 alias

# Check that it was added
ifconfig bridge100 | grep "inet "

Expected result:

inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet 10.10.0.1 netmask 0xffffff00 broadcast 10.10.0.255

or using a script

TARGET_BRIDGE=$(ifconfig -v | grep -B 20 "member: vmenet" | grep "bridge" | awk -F: '{print $1}' | head -n 1)

if [ -z "$TARGET_BRIDGE" ]; then
    echo "Error: bridge not found. Check if VM is running."
else
    echo "VM found on $TARGET_BRIDGE. Assigning 10.10.0.1..."
    sudo ifconfig $TARGET_BRIDGE 10.10.0.1/24 alias
fi

Creating cloud-init configuration for VM

Let’s create the following cloud-init configuration file that contains settings for the network 10.10.0.0/24

cat > multipass-static-ip.yaml << 'EOF'
#cloud-config

write_files:
  - path: /etc/netplan/60-static-ip.yaml
    permissions: '0600'
    content: |
      network:
        version: 2
        ethernets:
          default:
            dhcp4: true
            addresses:
              - 10.10.0.10/24
            routes:
              - to: default
                via: 10.10.0.1
                metric: 200

runcmd:
  - netplan apply

hostname: test-vm
EOF

To configure the network in Ubuntu, we use Netplan. We add the settings for it to the file /etc/netplan/60-static-ip.yaml. The file contents are located in the content: field. Note the line permissions: '0600', which sets read-write permissions for the root user only. If the permissions are too permissive, Netplan will notify you and will not apply the settings. The netplan apply command applies the settings from the /etc/netplan/ folder. The hostname field contains the name for our virtual machine, which will be added to the /etc/hostname file.

Launching VM and checking operation

Let’s create our virtual machine

# Create VM with cloud-init configuration
multipass launch --name test-vm --cloud-init multipass-static-ip.yaml

Multipass will create a virtual machine with the name specified in the --name/-n parameter and will use the cloud-init settings from the file specified in --cloud-init.

Let’s check the network settings of our virtual machine

# Check IP addresses on VM
multipass exec -n test-vm -- ip addr show enp0s1

# or use netplan
multipass exec -n test-vm -- netplan status

# Should show two IPs:
# - 192.168.2.x (DHCP)
# - 10.10.0.10 (static)

Now it’s time to check the connection

# From host to VM
ping -c 4 10.10.0.10

# From VM to host
multipass exec -n test-vm -- ping -c 4 10.10.0.1

# Check internet access from VM
multipass exec -n test-vm -- ping -c 4 8.8.8.8

✅ Static IP address works!

Technical details

# Output list of configuration files
multipass exec -n test-vm -- sudo ls -la /etc/netplan

# Get merged network configuration
multipass exec -n test-vm -- sudo netplan get

We see that in the /etc/netplan directory there are files 50-cloud-init.yaml, which is created by cloud-init during virtual machine initialization, and the file 60-static-ip.yaml, which we passed in the settings.

Running the sudo netplan get command will give us the merged network configuration. Compare it with the content of the 50-cloud-init.yaml file.

# Get content of 50-cloud-init.yaml
multipass exec -n test-vm -- sudo cat /etc/netplan/50-cloud-init.yaml

Pay attention to the name of the network interface that cloud-init uses. This is the one we used in our settings (not enp0s1).

network:
  version: 2
  ethernets:
    default: # network interface name
      match:
        macaddress: "52:54:00:ae:24:22"
      dhcp-identifier: "mac"
      dhcp4: true

Adding static IP address to the second network interface of the virtual machine

In addition to adding a static IP address to the first network interface, we can do this for other network interfaces of the virtual machine. For this, on the host, you need to add/create the corresponding network bridge.

Multipass bridge for the second network interface

Let’s launch a temporary VM to create a new bridge (bridge101)

multipass launch --name sandbox-vm --network name=en0,mode=manual

The parameter --network name=en0,mode=manual will create a new network interface enp0s2 in the virtual machine, which will be bound to a new bridge. However, after creation, this interface will be inactive because no IP address was assigned to it.

The (dis)advantage of this approach is that after deleting all virtual machines bound to this bridge, it will be automatically removed from the system. It exists as long as there are virtual machines bound to it.

This command allows you to find the name of this bridge:

ifconfig -v | grep -B 20 "member: vmenet" | grep "bridge" | awk -F: '{print $1}' | tail -n 1

Most likely the name will be bridge101.

Let’s add an alias to the bridge

# Add IP address to bridge
sudo ifconfig bridge101 10.10.1.1/24 alias

# Check that it was added
ifconfig bridge101 | grep "inet "

Expected result:

inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255

TARGET_BRIDGE=$(ifconfig -v | grep -B 20 "member: vmenet" | grep "bridge" | awk -F: '{print $1}' | tail -n 1)

if [ -z "$TARGET_BRIDGE" ]; then
    echo "Error: bridge not found. Check if VM is running."
else
    echo "VM found on $TARGET_BRIDGE. Assigning 10.10.1.1..."
    sudo ifconfig $TARGET_BRIDGE 10.10.1.1/24 alias
fi

Creating cloud-init configuration for the second network interface VM

Just like in the first case, let’s create a cloud-init configuration for setting up the virtual machine’s network interface.

cat > multipass-static-ip1.yaml << 'EOF'
#cloud-config

write_files:
  - path: /etc/netplan/60-custom-network.yaml
    permissions: '0600'
    content: |
      network:
        version: 2
        ethernets:
          enp0s2:
            addresses:
              - 10.10.1.20/24
            # WE REMOVE "via: 10.10.0.1" (default gateway)
            # Instead, we just allow direct access to the 10.10.1.0/24 network
            routes:
              - to: 10.10.1.0/24
                scope: link
runcmd:
  - netplan apply
EOF

Launching and checking the virtual machine operation

Let’s launch a virtual machine named test-vm1.

# Create a VM with cloud-init configuration
multipass launch --name test-vm1 --network name=en0,mode=manual --cloud-init multipass-static-ip1.yaml

Wait for the VM creation and launch process to complete. After that, we can delete our temporary virtual machine, which we used to have the system create a new network bridge.

multipass delete sandbox-vm --purge

Creating the test-vm1 virtual machine is the same as creating test-vm, with the difference that it will have two network interfaces.

Now let’s check the virtual machine’s network interface settings

# Check the IP addresses on the VM
multipass exec -n test-vm1 -- ip addr show

# or use netplan
multipass exec -n test-vm1 -- netplan status

You should see two IPs:

192.168.2.x (DHCP) on enp0s1
10.10.1.20 (static) on enp0s2

Let’s check the connection

# From host to VM1
ping -c 4 10.10.1.20

# From VM1 to host
multipass exec -n test-vm1 -- ping -c 4 10.10.1.1

# Traffic between VM and VM1
multipass exec -n test-vm1 -- ping -c 4 10.10.0.10
multipass exec -n test-vm -- ping -c 4 10.10.1.20

# Check Internet access from VM1
multipass exec -n test-vm1 -- ping -c 4 8.8.8.8

✅ The static IP address on the second network interface is working!

Cleaning up

Delete virtual machines with the command multipass delete <name-vm> --purge or all at once with multipass delete --all --purge.

After deleting all virtual machines that were attached to the bridge101 bridge, it will be removed automatically.

The result of executing ifconfig -v bridge101 will be

ifconfig: interface bridge101 does not exist

To remove the alias from bridge100, execute sudo ifconfig bridge100 -alias 10.10.0.1. You can also clear the cache with sudo arp -d -a.

Summary

With these instructions, you can add static addresses to Multipass virtual machines. This can be useful when you need to use a pool of pre-allocated addresses.

⚠️ This guide was created and tested for macOS. Working with other operating systems may differ depending on the features and approaches you use.

Deploying a Kubernetes Cluster on a Local Machine: A Complete Step-by-Step Guide

2025-12-12T06:30:00+00:00

💡 You won’t find such a detailed step-by-step guide in the official Kubernetes documentation! The official docs contain separate recommendations. Here everything is gathered in one place so you can quickly and easily create your first cluster.

What we’ll do

We’ll create a full Kubernetes cluster on your local machine. If you have separate physical machines, you can adapt this guide to run on physical hardware. For this guide we will use:

Multipass — tool to create Ubuntu virtual machines
kubeadm — primary tool to initialize the cluster
Flannel — CNI plugin to create Pod networking
MetalLB — load balancer for our cluster

Our cluster will consist of three nodes:

1 control plane node
2 worker nodes

Prerequisites

Before starting, install Multipass:

brew install multipass

Step 1: Create virtual machines

Define the list of nodes

Create an array with the names of the three virtual machines. This is just a simple list of the machines we need.

NODES=(k8s-control k8s-worker-1 k8s-worker-2)

Create VMs

Now use multipass to create three Ubuntu virtual machines.

for NODE in "${NODES[@]}"; do
  multipass launch --name $NODE --cpus 2 --memory 4G --disk 20G
done

Using the for NODE in "${NODES[@]}" loop we iterate over each name; multipass launch --name $NODE creates a VM with the given name and these parameters:

--cpus 2 — allocate 2 CPU cores (minimum for K8s)
--memory 4G — allocate 4 GB of RAM
--disk 20G — allocate 20 GB of disk space

Multipass will automatically download Ubuntu. This will take a few minutes ☕.

Step 2: Prepare all nodes

Once the virtual machines are created, start configuring them. These steps need to be executed on all three machines.

2.1. System update

echo "=== [1/7] Updating system on all nodes ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- bash -c "
    sudo apt-get update &&
    sudo apt-get upgrade -y
  "
done

multipass exec $NODE — runs a command on the VM
sudo apt-get update — updates the package lists
sudo apt-get upgrade -y — installs all upgrades; -y answers yes to prompts

It’s important to start with an up-to-date system with the latest security fixes and package updates.

After this command all packages will be updated to the latest versions.

2.2. Disable firewall

echo "=== [2/7] Disabling firewall on all nodes ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- sudo ufw disable
done

UFW is Ubuntu’s firewall. For a learning cluster we disable the firewall to avoid networking issues between nodes.

⚠️ In production you should configure the firewall properly and open required ports!

2.3. Load kernel modules

echo "=== [3/7] Configuring kernel modules ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- bash -c "
    echo -e 'overlay\nbr_netfilter' | sudo tee /etc/modules-load.d/k8s.conf
    sudo modprobe overlay
    sudo modprobe br_netfilter
  "
done

Kubernetes requires these two Linux kernel modules to be enabled:

overlay — for container filesystem layering
br_netfilter — for network connectivity between containers

The first line writes these modules to a config file so they’re loaded at boot. The next two lines load them now.

2.4. Configure sysctl networking parameters

echo "=== [4/7] Configuring networking sysctl parameters ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- bash -c "
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
    sudo sysctl --system
  "
done

We configure kernel networking settings:

bridge-nf-call-iptables — allows iptables to see bridged traffic (IPv4 and IPv6)
ip_forward — enables packet forwarding between interfaces

sysctl --system applies these settings immediately.

This is critical for Kubernetes networking!

2.5. Install containerd

echo "=== [5/7] Installing containerd ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- sudo apt-get install -y containerd
done

Containerd is the runtime that runs and manages containers. Kubernetes supports multiple runtimes; containerd is a recommended and popular choice.

2.6. Configure containerd

echo "=== [6/7] Configuring containerd and CRI ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- bash -c "
    sudo mkdir -p /etc/containerd
    containerd config default | sudo tee /etc/containerd/config.toml

    # Update the sandbox image
    sudo sed -i 's/registry.k8s.io\\/pause:3.8/registry.k8s.io\\/pause:3.10.1/' /etc/containerd/config.toml

    # Enable cgroup via systemd
    sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

    # Add crictl config
    sudo tee /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

    sudo systemctl restart containerd
    sudo systemctl enable containerd
  "
done

What happens here:

Generate the default containerd config into /etc/containerd/config.toml
Update the Kubernetes pause image to version 3.10.1
Enable SystemdCgroup to use systemd for cgroup management
Configure crictl to talk to containerd
Restart and enable containerd

2.7. Install Kubernetes components

echo "=== [7/7] Installing Kubernetes components ==="
for NODE in "${NODES[@]}"; do
  multipass exec $NODE -- bash -c "
    sudo apt-get install -y apt-transport-https ca-certificates curl gpg

    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key \
      | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

    echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' \
      | sudo tee /etc/apt/sources.list.d/kubernetes.list

    sudo apt-get update
    sudo apt-get install -y kubelet kubeadm kubectl
    sudo apt-mark hold kubelet kubeadm kubectl
    sudo systemctl enable kubelet
  "
done

Prepare tools for secure package installation (HTTPS, certificates, GPG)
Add the official Kubernetes signing key
Add the Kubernetes v1.34 repository
Install:
- kubelet — agent on each node
- kubeadm — initializer tool
- kubectl — CLI client

apt-mark hold prevents automatic updates; component versions must match across the cluster. Enable kubelet.

2.8. Done! ✅

These steps can be combined into a single script to prepare all nodes for cluster creation.

All three machines are now ready to become part of a Kubernetes cluster.¹

Step 3: Initialize the control plane

Connect to the control plane

multipass shell k8s-control

This opens a shell inside the k8s-control VM. Now we’re working directly on that machine.

Get the control plane IP

CONTROL_IP=$(hostname -I | awk '{print $1}')

hostname -I — shows all IP addresses of the machine
awk '{print $1}' — extracts the first address
$() — stores the result in CONTROL_IP

We need the IP so worker nodes know where to connect.

Initialize the cluster 🚀

sudo kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=$CONTROL_IP

kubeadm init initializes the control plane (API server, scheduler, controller-manager). At the end it prints a kubeadm join command to run on worker nodes.

Parameters:

--pod-network-cidr=10.244.0.0/16 — Pod network range (for Flannel)
--apiserver-advertise-address — IP the API server advertises

⏱️ This takes 1–2 minutes.

📝 Save the kubeadm join command output — you’ll need it for workers!

Configure kubectl

kubeadm also prints instructions to configure kubectl for the current user:

mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

kubectl requires a configuration file to connect to the cluster.

mkdir -p ~/.kube — create kube config directory
cp admin.conf ~/.kube/config — copy admin kubeconfig
chown — change ownership so kubectl can run without sudo

Step 4: Install CNI plugin — Flannel

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Flannel is a Container Network Interface (CNI) plugin that enables Pod networking across nodes.

Kubernetes does not provide cluster networking by itself; Flannel deploys the necessary DaemonSet, ConfigMap, ServiceAccount, and other resources.

Step 5: Join worker nodes

During kubeadm init you received a kubeadm join command. Run it on each worker node.

for NODE in k8s-worker-1 k8s-worker-2; do
  multipass exec $NODE -- sudo kubeadm join 192.168.2.26:6443 \
    --token bsw6fd.e7624wl2688fybjx \
    --discovery-token-ca-cert-hash sha256:7850aa1c6181277e284a08b81256979db25698a89982f0885540376a5376e0bd
done

⚠️ IMPORTANT: In your case the IP, token and hash will be different! Use the command that kubeadm init printed.

What this does:

multipass exec $NODE -- — runs the join command on each worker
192.168.2.14:6443 — API server address (port 6443)
--token — temporary token generated by kubeadm
--discovery-token-ca-cert-hash — CA cert SHA256 hash to verify authenticity

Step 6: Verify the cluster

Return to the control plane node and check node status.

multipass shell k8s-control
kubectl get nodes

You should see three nodes with STATUS Ready:

NAME            STATUS   ROLES           AGE   VERSION
k8s-control     Ready    control-plane   5m    v1.34.0
k8s-worker-1    Ready    <none>          2m    v1.34.0
k8s-worker-2    Ready    <none>          2m    v1.34.0

If a node shows NotReady, wait a minute — Flannel may still be provisioning ⏳

Step 7: Install MetalLB

What is MetalLB?

MetalLB is a load balancer implementation for bare-metal clusters. In cloud-managed Kubernetes, LoadBalancer services are provided by the cloud provider. For a local cluster, MetalLB provides similar functionality.

Apply manifests

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml

Wait for readiness

kubectl wait --namespace metallb-system \
  --for=condition=ready pod \
  --selector=app=metallb

This waits until all MetalLB pods are ready.

Label worker nodes

To label all nodes whose names start with k8s-worker-, get node names and apply a label to each.

NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}' | tr -s '[[:space:]]' '\n' | grep '^k8s-worker-')

for NODE in $NODES; do
  echo "Applying label to node: $NODE"
  kubectl label node "$NODE" metallb-role=worker --overwrite
done

We add the label metallb-role=worker to worker nodes. Labels help select resources in Kubernetes. MetalLB will run only on nodes with this label.

Configure IP pool and L2Advertisement

CONTROL_IP=$(hostname -I | awk '{print $1}')
BASE_IP=$(echo $CONTROL_IP | cut -d. -f1-3)

cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - ${BASE_IP}.200-${BASE_IP}.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: worker-nodes-l2
  namespace: metallb-system
spec:
  ipAddressPools:
  - first-pool
  nodeSelectors:
  - matchLabels:
      metallb-role: worker
EOF

What this does:

Get the base IP from the control plane IP (e.g. if control IP is 192.168.2.14, BASE_IP becomes 192.168.2)
IPAddressPool defines a range of external IPs MetalLB can allocate (.200–.250 in this example)
L2Advertisement configures MetalLB to announce those IPs via ARP (Layer 2)
nodeSelectors restrict MetalLB to nodes labeled metallb-role=worker

Step 8: Demo 🎉

Create a Deployment

kubectl create deployment hello \
  --image=nginxdemos/hello:plain-text \
  --replicas=3 \
  --port=80

Creates 3 replicas of a simple nginx demo app.

deployment — manages a set of identical Pods
--image — Docker image for the containers
--replicas=3 — run 3 copies
--port=80 — container port

Create a LoadBalancer Service

kubectl expose deployment hello \
  --type=LoadBalancer \
  --port=80

A LoadBalancer service will get an external IP from MetalLB.

expose deployment — creates a Service for the Deployment
--type=LoadBalancer — request an external IP (provided by MetalLB)
--port=80 — service port

Test load balancing

EXTERNAL_IP=$(kubectl get svc hello -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "LB IP: $EXTERNAL_IP"

for i in {1..10}; do
  echo "Request $i:"
  curl -s http://$EXTERNAL_IP | grep "Server address"
  echo "---"
done

This obtains the external IP allocated by MetalLB and makes 10 requests, showing which Pod served each request.

Expected result: requests are distributed across the three Pods.

Request 1:
Server address: 10.244.1.2:80
---
Request 2:
Server address: 10.244.2.3:80
---
Request 3:
Server address: 10.244.1.4:80
---

🚨 Important notes about using SWAP

By default Multipass VMs have swap disabled.

kubelet refuses to run if swap is enabled because swapping can cause unpredictable Pod behavior (containers being pushed to disk, slowing them down).

In our case swap is disabled by default — so it’s fine! ✅

If you deploy Kubernetes on systems where swap is enabled:

Option 1: Turn off swap

sudo swapoff -a
# And comment out swap in /etc/fstab

Option 2: Configure kubelet to run with swap support (experimental in K8s 1.28+)

Useful commands

Status checks

kubectl get nodes
kubectl get pods -A
kubectl get svc

View logs

kubectl logs -n kube-system -l app=flannel
kubectl logs -n metallb-system -l app=metallb

Delete resources

kubectl delete svc hello
kubectl delete deployment hello

Stop the cluster

# Stop all VMs
for NODE in "${NODES[@]}"; do
  multipass stop $NODE
done

# Start again
for NODE in "${NODES[@]}"; do
  multipass start $NODE
done

Remove the cluster

for NODE in "${NODES[@]}"; do
  multipass delete $NODE
done
multipass purge

Summary

Congratulations! 🎉 You just created a full Kubernetes cluster on your local machine.

What we did:

✅ Created 3 virtual machines
✅ Configured containerd and kernel modules
✅ Initialized the control plane
✅ Joined worker nodes
✅ Installed Flannel for networking
✅ Configured MetalLB for LoadBalancer services
✅ Deployed a test app with load balancing

Now you have a local environment for:

experimenting with Kubernetes
testing manifests
learning cluster architecture
preparing for certifications (CKA, CKAD)

Hope this guide was useful!

If you have questions or issues — leave a comment. Share this post with colleagues who may find it helpful 🚀

Happy Kubernetes! ☸️

Disclaimer 🚨

This cluster is intended for development and testing.

For production use you need additional configuration:

Firewall and Network Policies
RBAC (Role-Based Access Control)
Secrets management
Monitoring and logging
etcd backups
High Availability control plane
Vulnerability scanning

Preparation script https://gist.github.com/Andygol/37d1397423e535bd0f7fabb593e81c41 ↩

From Docker to Kubernetes: When Containers Stop Being Simple

2025-10-30T08:30:00+00:00

Wow, you’ve finally done it!
After days or even weeks of work — your app is fully Dockerized.

You’ve got a Node.js API, a React frontend, and a Postgres database, all wrapped up nice and neat in their own little containers.
One docker-compose up, and everything comes alive.
Your local setup feels like an orchestra — every container plays its part, and you’re the conductor.

You feel proud. A real DevOps wizard.

Then Comes Production

Everything runs perfectly on your laptop.
But once it’s time to deploy to real servers, the illusion of simplicity is gone.

You want reliability.
You need scalability.
And it seems obvious:

“I’ll just run these containers on a few servers.”

Simple? Not really.

When Containers Turn Into Chaos

You hit your first wall:

How will the frontend find the API when container IPs keep changing?
What happens when a server crashes at 4 AM? Who restarts the containers?
How can you update the API image without taking everything down?

You start writing Bash scripts, copying images via SSH, and trying to balance traffic manually.
But every update feels risky.
Every crash — a small disaster.

Your once-elegant Docker setup slowly turns into a fragile web of scripts and hope.

Enter Kubernetes: Not Just Docker on Steroids

Enter Kubernetes (or simply K8s)¹.

You’ve probably heard about it — maybe you think it’s overly complicated or something only big tech companies use.
And yes, it is complex at first.
But that’s because Kubernetes solves a completely different problem.

Docker helps you package an application. Kubernetes helps you run and manage those applications at scale.

It doesn’t just start containers — it manages their entire lifecycle: deployment, scaling, self-healing, updates, and service discovery.

And it all begins with a change in mindset.

Declarative Thinking: Focus on “What,” Not “How”

In the Docker world, you act imperatively:

“Start this container here. Stop that one there.”

In Kubernetes, you act declaratively:

“I want three replicas of my API running image v1.2.
Each should have 500 MB of RAM.
They should all be reachable via api-service.”

You don’t tell the system how to do it.
You just describe what the desired end state should look like.

Kubernetes constantly watches the actual state of the system and works to make it match your desired state — automatically.

How Kubernetes Solves Real Production Problems

Automated Scheduling & Bin Packing

Kubernetes sees all your servers (called nodes) and decides where to run containers based on available resources.
It distributes workloads intelligently — no manual assignments needed.

Self-Healing

If a container crashes or a node fails, Kubernetes immediately detects it.
Desired state: 3 replicas.
Actual state: 2 replicas.
It spins up a new one.

No late-night SSH sessions, no manual restarts.
The system heals itself.

Horizontal Scaling

Traffic spikes? No problem.
You just update one line in your YAML:

spec:
  replicas: 12

Kubernetes launches more containers and spreads them across available nodes.

Don’t want to do it manually?
Enable autoscaling, and K8s will automatically adjust replica counts based on CPU or memory usage.

Service Discovery & Load Balancing

Containers don’t rely on IPs to find each other.
You create a Service — an abstraction that gives your app a stable name (like api-service) and an internal IP.

When the frontend calls api-service, Kubernetes automatically routes the request to one of the healthy API instances.
Traffic is balanced automatically.

No more hardcoded IPs. No more fragile networking hacks.

Automated Rollouts & Rollbacks

Need to update your API to version v1.3?
Just change the image tag in your YAML.

Kubernetes performs a rolling update — gradually spinning up new v1.3 containers while shutting down the old v1.2 ones.
No downtime. No user impact.

And if something goes wrong?
Kubernetes automatically rolls back to the previous stable version.

Kubernetes as the Operating System for Your Applications

Kubernetes isn’t just another DevOps tool.
It’s an operating system for distributed systems.

It handles resource management, updates, load balancing, recovery — everything that used to require dozens of scripts and sleepless nights.

You no longer waste time babysitting servers.
You can focus on what really matters — building great software.

Manageability

Kubernetes doesn’t promise simplicity — it promises control.
It lets you describe how your system should look, and then it handles everything else: placement, recovery, scaling, networking, and updates.

Docker was the first step.
Kubernetes is the next level of infrastructure maturity.

Containers made development easier.
Kubernetes makes production predictable.

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Services, support, and tools are widely available.
https://andygol-k8s.netlify.app/docs/concepts/overview/ ↩

Minute Changes in OpenStreetMap Aren’t That Minute

2025-05-08T08:30:00+00:00

OpenStreetMap is a great example of a collaborative project for building an open geospatial data repository. Collecting the data is only part of the process. It’s collected so that it can be used. You can obtain data for the current map area using the Export menu.

However, the ability to download data for the current area has some limitations. You won’t be able to do it if you are viewing the map at zoom level z11 or lower 👇. See the message in the yellow rectangle?

Planet OSM

If you need data for a larger area or even for the entire planet, head over to where the project offers data for download — the planet dump, https://planet.openstreetmap.org/

Here you can download data for the whole planet, either as a data snapshot for a specific date or as a full history dump, which are generated weekly.

This data is provided free of charge by the project, but you can support the project by donating or becoming a member of the OpenStreetMap Foundation.

Diff Files

For those less familiar with the project, diffs are sets of changes that occurred in the data and are published by OpenStreetMap every day, hour, and minute. These files list all changes made to the database within a selected time period.

Developers can use these diffs to update their local copies of the database, which are then used for geocoding, map rendering, and other tools in near real-time.

One of the key features that enables tracking changes in this massive database is the concept of “minute diffs”. At first glance, the name suggests almost instant updates of every change made. But is that really the case? Sounds ideal, right? But there are a few nuances worth considering:

“Near” real time is the key phrase. Although change files are generated every minute, their actual availability may be delayed. Depending on server load, the generation and publication process may take a bit longer. So, while you may see changes within a few minutes, it won’t always be exactly one minute.
Processing diffs is not instant. Getting the diff file is only the first step. Then the software must download, parse, and apply the changes to the local database copy. This process also takes some time, depending on the size of the change file and your hardware’s performance. During peak editing times on OpenStreetMap, these files can become significantly larger, slowing down the update process.
Applying the diffs in the right order is essential. To ensure data consistency, changes must be applied in the correct order. This means if you miss several minute diffs (e.g., due to network issues), you’ll need to process them sequentially before your database is up to date again.

Using Diffs

You’ve obtained the planet dump and loaded it into your map rendering or routing service. That’s just the beginning. In today’s fast-paced world, information changes rapidly, and those who keep up with those changes will have a significant advantage.

The planet dump file in OpenStreetMap is generated weekly.

As of this writing, May 8, 2025, planet-latest.osm.bz2 was created on 2025-05-02 23:50 and its size was 150G; you can also get the data in binary pbf format — planet-latest.osm.pbf, 2025-05-02 23:50, 81G; planet-250428.osm.pbf, 2025-05-02 23:50, 81G.

This latest dump contains data as of April 25, 2025, and was published on May 2, 2025.

This means that at the time of publication, the dump is already 7 days out of date. Add a couple more days before you get around to downloading it, and your data may be 1–2 weeks outdated at the start of deployment. Then add the time needed to load the dump into your system and you might find that your data is already 3–5 weeks stale by the time you begin using it (when talking about the full-planet scale). To catch up and get current data, you should use replication files: https://planet.openstreetmap.org/replication/, our daily, hourly, and minute diffs.

Replication Process

Each diff file is accompanied by an additional metadata file describing the diff — state.txt

The state.txt file contains the timestamp and the sequence number of the diff. Using this number, you can retrieve the corresponding diff file (717.osc.gz, 2025-05-08 17:01, 101K). For convenience, the sequence number is split into triplets: 006/590/717.

The latest diff information is available in the root of the respective replication type. For minute diffs, this is: https://planet.openstreetmap.org/replication/minute/state.txt.

To synchronize your local data with the current OpenStreetMap data, you need to determine the last timestamp present in your local dump. For example, in a dump dated March 24, 2025, the last timestamp was 2025-03-24T00:59:53Z.

Now that you have this timestamp, you need to find the corresponding diff file (daily or hourly) and its number. This can help:

https://replicate-sequences.osm.mazdermind.de/?2013-01-01T10:00:00Z

curl "https://replicate-sequences.osm.mazdermind.de/?$(date -u -d "@$(stat -c "%Y" planet-latest.osm.pbf)" +"%FT%TZ")"

Finding Diffs Manually

For daily and hourly diffs, you can also use next approach:

Convert the initial timestamp (from the dump) to Unix epoch time:

date -d "2025-03-24T00:59:53Z" +%s
# or explicitly specify the format
date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "2025-03-24T00:59:53Z" +%s

This gives you the Unix time in seconds: 1742777993.

Retrieve the latest state.txt timestamp and sequence number (daily or hourly)
Calculate the time difference (in days or hours), subtract this value from the sequence number, and get the download link for the diff file from which to begin syncing your local data

DUMP_EPOCH_TS=$(date -d "2025-03-24T00:59:53Z" +%s)
REFERENCE_DATE=$(wget -q -O - https://planet.openstreetmap.org/replication/day/state.txt 2>/dev/null | grep "timestamp" | cut -d'=' -f2 | sed 's/T/ /;s/Z//; s/\\//g')
REFERENCE_SEQ=$(wget -q -O - https://planet.openstreetmap.org/replication/day/state.txt 2>/dev/null | grep "sequenceNumber" | cut -d'=' -f2 )
LAST_EPOCH_TS=$(date -d "$REFERENCE_DATE" +%s)

# Example for calculating difference in days (86400 seconds = 24 hours)
# Use 3600 instead for hourly diffs
DIFF_TS=$(( ($LAST_EPOCH_TS - DUMP_EPOCH_TS) / 86400 ))
TARGET_SEQ=$(( (10#$REFERENCE_SEQ - $DIFF_TS) + 1 ))
seq_padded=$(printf "%09d" "$TARGET_SEQ")
url="https://planet.openstreetmap.org/replication/day/${seq_padded:0:3}/${seq_padded:3:3}/${seq_padded:6:3}.state.txt"

The $url variable will contain a link to the state.txt file needed to begin the replication process. This file is required for osmosis or similar tools. (For more details on the replication process, see: https://wiki.openstreetmap.org/wiki/Planet.osm/diffs)

The approach described above works well for daily and hourly diffs but does not work for minute diffs.

Issues with Minute Diff Calculations

The approach described above for calculating the diff file number to begin replication does not work for minute diffs. Why? Because minute diffs can take more than 60 seconds to generate, resulting in “slippage” in the timeline.

For example, if you take the range of minute diffs from 6590000 to 6590227, you get 233 minutes but only 227 diff files.

On the charts, you can see how long it took to generate each minute diff and which minutes they were created in. You can observe that at 7:59, 8:02, and 8:05 no diffs were created — slippage occurred.

Sequence	Timestamp	Epoch, seconds	Epoch, minutes	Time, sec	Clock
6590194	2025-05-08 8:12:08	1746681128	29111352	65	8:12
6590193	2025-05-08 8:11:05	1746681065	29111351	63	8:11
6590192	2025-05-08 8:10:01	1746681001	29111350	64	8:10
6590191	2025-05-08 8:09:01	1746680941	29111349	60	8:09
6590190	2025-05-08 8:08:01	1746680881	29111348	60	8:08
6590189	2025-05-08 8:07:02	1746680822	29111347	59	8:07
6590188	2025-05-08 8:06:00	1746680760	29111346	62	8:06
6590187	2025-05-08 8:04:59	1746680699	29111344	61	8:04
6590186	2025-05-08 8:04:00	1746680640	29111344	59	8:04
6590185	2025-05-08 8:03:00	1746680580	29111343	60	8:03
6590184	2025-05-08 8:01:59	1746680519	29111341	61	8:01
6590183	2025-05-08 8:00:58	1746680458	29111340	61	8:00
6590182	2025-05-08 8:00:00	1746680400	29111340	58	8:00
6590181	2025-05-08 7:58:56	1746680336	29111338	64	7:58
6590180	2025-05-08 7:57:51	1746680271	29111337	65	7:57

This table also shows that there were 3 ‘slippages’ within 15 minutes.

Is this critical for everyday use? Since this has been happening for a long time, it seems that it is not. Existing change handling tools don’t care about this. Yes, it can cause excessive traffic if you go further back in time than necessary for minute diffs, but it won’t affect the final result. You will get your own dataset that will be synchronised with OpenStreetMap data with a lag of up to 2 minutes.

Can this be fixed? Maybe.

Processing time-based data is always a non-trivial task. To fix this situation:

The timestamp field of the diff metadata should contain the start time (the moment in time when the diff creation process started), which should ideally be an integer (or very close to it) time in minutes, hours, and days:
- 13:00:00, 13:01:00, 13:02:00 for minutes;
- 14:00:00, 15:00:00 - for hours; and
- 00:00:00 for days.
The end time of the diff creation process will be the time of the modification of the file itself, although it can also be specified in the metadata of the change set (this is the time currently contained in timestamp).
The creation of the next diff should start at the specified time, regardless of whether the previous diff has finished. For some time, the process of creating them can take place in parallel.
If there are no changes, create an empty diff file to maintain a monotonous order of change numbering so that there are no gaps in the time series.

If you are annoyed with this situation, you can submit suggestions for fixing it to the project maintainers, and it may be fixed. It’s hard to say how soon.

Recap

Diffs remain an extremely valuable tool for keeping local copies of OpenStreetMap data up to date. However, it is important to understand their limitations and not to take the name literally, especially with minute diffs.

For developers who use minute diffs, this means that they need to have a reliable infrastructure for processing them, take into account possible delays, and be prepared to process large amounts of data during periods of active editing.

For ordinary users, this is just an interesting fact about how the open source map they love works behind the scenes. The next time you make a small edit and don’t see it on your map server instantly, don’t worry — your ‘minute’ diff might just be on its way!

What do you think of minute diffs? Have you experienced any delays in data updates? Share your experience in the comments!

Getting access to the host file system for Persistent Volume in Kind

2025-04-05T08:30:00+00:00

Kubernetes is no longer something that only platform engineers work with. More and more applications are wrapped in containers and run in container environments. What if, for some reason, you don’t have access to a cloud platform but still need to develop an application that will run in the cloud? You can use Kind to deploy Kubernetes locally.

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. It was originally designed for testing Kubernetes itself, but it can also be used for local development or CI.

To run Kind, you’ll need docker, podman, or another container engine. Refer to the Kind Quick Start Guide on the official site.

Creating a Cluster

So, we’ve installed Kind and a container runtime. We also have kubectl – the command-line tool for interacting with the cluster.

To create a local cluster, use the kind create cluster command.

You can always get the necessary help by using kind [command] --help.

We’ve created a local cluster named kind-kind. This is the default name Kind uses if the -n cluster_name or --name cluster_name parameter is not specified.

Instead of just passing CLI flags, you can also use a manifest to create a cluster with the desired parameters.

cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: my-super-cluster
nodes:
- role: control-plane
- role: worker
  extraMounts:
  - hostPath: /path/to/local/data
    containerPath: /data
# - role: worker
# - role: worker
#   extraMounts:
#   - hostPath: /path/to/local/data/dump
#     containerPath: /data/dump
#   - hostPath: /path/to/local/data/diff
#     containerPath: /data/diff

☝️ Here you can specify the number of desired nodes, their role, and—most importantly in our case—the local file system path on the host to be mounted into the cluster nodes and used as backing storage for our Persistent Volumes. See Extra Mounts in the Kind documentation.

Let’s apply our config to create the cluster

kind create cluster --config kind-config.yaml

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.32.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-my-super-cluster"
You can now use your cluster with:

kubectl cluster-info --context kind-my-super-cluster

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Let’s verify that the host file system is mounted in the worker node of our cluster.

docker container inspect osm-cluster-worker \
  | jq '[{"Name": .[0].Name,
          "BindMounts": (
            .[] |
            .Mounts[] |
            select(.Type == "bind")
        )}]'

And we see everything is OK—the file system is mounted.

[
  {
    "Name": "/my-super-cluster-worker",
    "BindMounts": {
      "Type": "bind",
      "Source": "/host_mnt/path/to/local/data",
      "Destination": "/data",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    }
  },
  {
    "Name": "/my-super-cluster-worker",
    "BindMounts": {
      "Type": "bind",
      "Source": "/lib/modules",
      "Destination": "/lib/modules",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    }
  }
]

Creating a PersistentVolume and PersistentVolumeClaim

Let’s define a manifest for our Persistent Volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-super-cluster-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  hostPath:
    path: "/data"
  storageClassName: my-storageclass

We’ll also create a PersistentVolumeClaim to mount the PV in workloads:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name:  my-super-cluster-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: my-storageclass

Now the most important part 🥁—creating a StorageClass that explicitly links the PVC to the PV.

Note: Kind creates a default StorageClass when the cluster is created, but it has reclaimPolicy: Delete, which is not what we want.

kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  80m

This means the contents of the volume will be deleted once it is unmounted—something we want to avoid.

Let’s define our own StorageClass:

kubectl apply -f -  <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-storageclass
provisioner: rancher.io/local-path
parameters:
  nodePath: /data
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF

storageclass.storage.k8s.io/my-storageclass created

and check it:

kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
my-storageclass      rancher.io/local-path   Retain          WaitForFirstConsumer   false                  5m27s
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  91m

Make it the default, just in case:

kubectl patch storageclass my-storageclass -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

And make standard non-default:

kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Check the result:

kubectl get storageclass

NAME                        PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
my-storageclass (default)   rancher.io/local-path   Retain          WaitForFirstConsumer   false                  12m
standard                    rancher.io/local-path   Delete          WaitForFirstConsumer   false                  98m

Using the PersistentVolumeClaim in a Pod

Apply the PV and PVC manifests to the cluster:

kubectl apply -f pv.yaml -f pvc.yaml

Now create a pod that uses the PVC:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
spec:
  containers:
  - name: debug-container
    image: busybox:latest
    command: ["sh", "-c", "sleep 3600"]
    volumeMounts:
    - mountPath: "/data"
      name: my-super-cluster
  volumes:
  - name: my-super-cluster
    persistentVolumeClaim:
      claimName: my-super-cluster-pvc
EOF

Check that the PVC is bound to the PV and used by our test pod:

kubectl get pv

NAME                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
my-super-cluster-pv   100Gi      RWO            Retain           Bound    default/my-super-cluster-pvc   my-storageclass   <unset>                          9m20s

kubectl get pvc

NAME                   STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
my-super-cluster-pvc   Bound    my-super-cluster-pv   100Gi      RWO            my-storageclass   <unset>                 8m48s

The Bound status confirms that the PVC has successfully bound to the PV.

kubectl describe pod

Name:             debug-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.20.0.4
Start Time:       Fri, 04 Apr 2025 18:17:09 +0300
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:               10.244.0.5
IPs:
  IP:  10.244.0.5
Containers:
  debug-container:
    Container ID:  containerd://d030a6edfc13c314853f22efc505990bbbb8e3954ed1c9887b9c7b3be575a0be
    Image:         busybox:latest
    Image ID:      docker.io/library/busybox@sha256:37f7b378a29ceb4c551b1b5582e27747b855bbfaa73fa11914fe0df028dc581f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      sleep 3600
    State:          Running
      Started:      Fri, 04 Apr 2025 18:17:13 +0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from my-super-cluster (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wdzj (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  my-super-cluster:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  my-super-cluster-pvc
    ReadOnly:   false
  kube-api-access-5wdzj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m36s  default-scheduler  Successfully assigned default/debug-pod to kind-control-plane
  Normal  Pulling    8m36s  kubelet            Pulling image "busybox:latest"
  Normal  Pulled     8m32s  kubelet            Successfully pulled image "busybox:latest" in 3.395s (3.395s including waiting). Image size: 1855985 bytes.
  Normal  Created    8m32s  kubelet            Created container: debug-container
  Normal  Started    8m32s  kubelet            Started container debug-container

Our pod has been successfully created and is running.

Access the pod’s terminal and verify that the volume is mounted and functioning:

kubectl exec -it debug-pod -- sh

/ # ls -l / | grep data
drwxr-xr-x    2 root     root          4096 Apr  4 15:17 data
/ # touch /data/somefile.txt
/ # ls -l /data
total 0
-rw-r--r--    1 root     root             0 Apr  4 15:31 somefile.txt
/ #
/ # exit

Now check the host file system mounted into the cluster node—you should see the newly created somefile.txt.

Summary

We’ve created a Persistent Volume Claim to use in a workload, bound to a Persistent Volume via a custom StorageClass. The PV uses the file system of a cluster node, which in turn maps to the host file system.

This setup allows us to reliably store and reuse data in Persistent Volumes across workloads—even though workloads have an inherently ephemeral lifecycle. It also allows us to preload data from the host file system and make it available to pods.

Cleanup

To delete the cluster, run:

kind delete cluster --name kind-my-super-cluster

Deleting cluster "kind-my-super-cluster" ...

Wait for Kind to delete the cluster. If needed, manually remove the mounted files from the host file system.

Deploy a container from the ghcr.io private registry to Amazon ECS

2024-06-26T08:30:00+00:00

Everyone reaches a point where you need to deploy a container from a private repository to ECS. Amazon’s documentation isn’t the epitome of clarity, so I’ve prepared a guide for you (and primarily for myself) on what needs to be done.

Prerequisites

Container in ghcr.io

First, you need to upload your container to the GitHub Container Registry. This can be a public or private repository.

Log in to GitHub and navigate to the repository page where your container is located. In the Packages section, select the artifact that is your previously created container.

To access your registry, you will need a Private Access Token (PAT). To create a PAT, go to your account settings.

At the bottom of the page, select Developer settings.

Next, select Personal access tokens/Tokens (classic)/Generate new token (classic).

Alternatively, follow this link to create a PAT — https://github.com/settings/tokens/new.

Add a note if needed, choose the expiration date for the token, and specify the required permissions — in this case, we only need permissions to fetch packages (read:packages — Download packages from GitHub Package Registry). At the bottom of the page, click Generate token.

After that, you will see a page with your new token. Copy it and save it in a secure place. We will use it to access the registry.

ECS

You should have an AWS account set up, optionally with AWS CLI and Amazon ECS CLI installed locally. If you don’t want to install them locally, you can use AWS CloudShell.

Creating a CMK in AWS KMS

First, we need to create a CMK (Customer Master Key) and an alias for it in AWS KMS. The CMK is used by AWS Secret Manager for envelope encryption of data containing sensitive information. An alias acts as a name for your CMK, making it easier to remember and use than the key ID itself. You can also use the alias in your code. You can change the key in the future (to another one) while keeping the alias unchanged.

aws kms create-key --query KeyMetadata.Arn --output text

In response, you will receive the key ID in the form of an Amazon Resource Name (ARN):

arn:aws:kms:eu-central-1:123456789012:key/abc123de-4567-89fa-0bcd-efgh12345678

Now we will create an alias for our key:

aws kms create-alias \
  --alias-name alias/ecs-ghcr \
  --target-key-id arn:aws:kms:eu-central-1:123456789012:key/abc123de-4567-89fa-0bcd-efgh12345678

If you haven’t set up AWS CLI locally, you can use CloudShell and run all the commands in the command line there.

You can create the CMK in the AWS Console by going to AWS KMS and clicking the Create key button.

You will need the ARN of the CMK when creating the trust policy document in the next step.

Creating a Secret in AWS Secrets Manager

At this stage, we need to create a Secret that will store your login and password (access code) encrypted with the CMK for pulling your container image from a private registry.

aws secretsmanager create-secret \
  --name ghcr_io_pat \
  --description "Secret to get packages from ghcr.io" \
  --kms-key-id alias/ecs-ghcr \
  --secret-string '{"username":"your_nickname", "password":"ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}'

You should receive the following in response:

{
    "ARN": "arn:aws:secretsmanager:eu-central-1:123456789012:secret:ghcr_io_pat-abcdEF",
    "Name": "ghcr_io_pat",
    "VersionId": "4b43b832-df4c-48b3-b59a-bb18287e6c15"
}

The ARN of the secret should be in the output ☝️ of the previous command — in the ARN field. You will need to refer to this ARN when creating the trust policy document in the next step.

Creating an IAM Role for Task Execution

If you already have the ecsTaskExecutionRole, you can skip this step.

First, you need to create a trust policy document to specify the principal that will assume the role, which in this case is ECS task:

cat << EOF > ecs-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Create the role using the AWS CLI and the ecs-trust-policy.json file with the role description.

aws iam create-role \
  --role-name ecsTaskExecutionRole \
  --assume-role-policy-document file://ecs-trust-policy.json

To add basic permissions to other AWS service resources needed to run Amazon ECS tasks, attach the AWS ECS task execution role policy to the newly created role:

aws iam attach-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Now create a permissions policy document that allows the ECS task to decrypt and retrieve the secret created in AWS Secrets Manager.

cat << EOF > ecs-secret-permission.json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "arn:aws:secretsmanager:eu-central-1:123456789012:secret:ghcr_io_pat-abcdEF",
        "arn:aws:kms:eu-central-1:123456789012:key/abc123de-4567-89fa-0bcd-efgh12345678"
      ]
    }
  ]
}
EOF

☝️ Specify your Secret and CMK in the "Resource": [ <Secret>, <CMK> ] values.

Finally, add the inline permissions policy that allows your task to fetch ghcr.io username and password from AWS Secrets Manager. Note that you refer to the permissions policy document created in the previous step. Change the file path as needed to specify the correct file location:

aws iam put-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-name ECS-SecretsManager-Permission \
  --policy-document file://ecs-secret-permission.json

Configuring ECS CLI (Optional)

The Amazon ECS Command Line Interface (ECS CLI) provides commands to simplify the creation of an Amazon ECS cluster and the AWS resources required to set it up. After installing the ECS CLI, you can further configure your AWS credentials in a named ECS profile. Profiles are stored in the ~/.ecs/credentials file.

ecs-cli configure profile \
  --access-key <AWS_ACCESS_KEY_ID> \
  --secret-key <AWS_SECRET_ACCESS_KEY> \
  --profile-name <PROFILE_NAME>

You can also specify a default profile to use for all ECS CLI commands:

ecs-cli configure profile default --profile-name <PROFILE_NAME>

If you do not configure an ECS profile or set environment variables, the default AWS profile will be used. The access key and secret access key values can be viewed in the AWS Management Console.

You can further configure the ECS cluster name, launch type, and AWS region to use with the ECS CLI using the ecs-cli configure command. The <LAUNCH_TYPE> variable can be set to FARGATE or EC2.

ecs-cli configure \
  --cluster <CLUSTER_NAME> \
  --default-launch-type <LAUNCH_TYPE> \
  --config-name <CONFIG_NAME> \
  --region <AWS_REGION>

These values can also be specified or overridden using command flags in the subsequent steps.

Creating an Amazon ECS Cluster

Create an Amazon ECS cluster using the ecs-cli up command, specifying the cluster name, AWS region (e.g., eu-central-1), and FARGATE as the launch type:

ecs-cli up \
  --cluster <CLUSTER_NAME> \
  --region eu-central-1 \
  --launch-type FARGATE

By using the FARGATE launch type, AWS Fargate manages the compute resources on your behalf, so you do not need to specify your own EC2 container instances. By default, the ECS CLI will also create an AWS CloudFormation stack to create a new VPC with an attached internet gateway, 2 public subnets, and a security group. You can also specify your own resources using flags in the command above.

Configuring Security Group

After successfully creating the ECS cluster, you should see the VPC and subnet IDs displayed in the terminal. Next, get the JSON description of the newly created security group and note the security group ID or GroupId. Replace the <VPC_ID> variable with the ID of the newly created VPC.

aws ec2 describe-security-groups \
  --filters Name=vpc-id,Values=<VPC_ID> \
  --region eu-central-1

In response, you should get an output similar to this one:

{
    "SecurityGroups": [
        {
            "Description": "default VPC security group",
            "GroupName": "default",
            "IpPermissions": [
                {
                    "IpProtocol": "-1",
                    "IpRanges": [],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "UserIdGroupPairs": [
                        {
                            "GroupId": "sg-04512c8a7bff9b34e",
                            "UserId": "123456789012"
                        }
                    ]
                },
                …
            ],
            …
        }
    ]
}

Add an inbound rule to the security group to allow HTTP traffic from any IPv4 address. Replace the <SG_ID> variable with the GroupId obtained in the previous step. This inbound rule will allow you to verify that your server is running in your task and that the image from the private registry was successfully pulled from GHCR.

aws ec2 authorize-security-group-ingress \
  --group-id <SG_ID> \
  --protocol tcp \
  --port 8080 \
  --cidr 0.0.0.0/0 \
  --region eu-central-1

Creating an Amazon ECS Service

Amazon ECS service allows you to run and maintain a specified number of instances of a task definition simultaneously. The ECS CLI allows you to create a service using a Docker compose file. Create the following docker-compose.yml file that defines a container that serves port 8080 for incoming traffic to the server. To reference an image stored in your private registry in GHCR, replace the <USER_NAME> variable with your GitHub username, the <REPO_NAME> variable with your private repository name, and the <TAG_NAME> variable with the tag you used.

cat << EOF > docker-compose.yml
version: "3"
services:
    web:
        image: ghcr.io/<USER_NAME>/<REPO_NAME>:<TAG_NAME>
        ports:
            - 8080:8080
EOF

Thus, if you try to deploy the image from the example at the beginning, the value of the image key will be as follows: ghcr.io/andygol/switch2osm-mkdocs:main.

You will also need to create the following ecs-params.yml file to specify additional parameters for your service specific to Amazon ECS. Note that the services field below corresponds to the services field in the Docker Compose file above, corresponding to the name of the container to run. When the ECS CLI creates a task definition from the docker-compose.yml file, the web fields will be merged into the ECS container definition, including the container image it will use and the GHCR repository credentials it will need to access it. Replace the <SECRET_ARN> variable with the ARN of the AWS Secrets Manager secret you created earlier. Replace the <SUB_1_ID>, <SUB_2_ID>, and <SG_ID> variables with the IDs of the 2 public subnets and the security group that were created along with the ECS cluster.

cat << EOF > ecs-params.yml
version: 1
task_definition:
  task_execution_role: ecsTaskExecutionRole
  ecs_network_mode: awsvpc
  task_size:
    mem_limit: 0.5GB
    cpu_limit: 256
  services:
    web:
        repository_credentials: 
            credentials_parameter: "<SECRET_ARN>"
run_params:
  network_configuration:
    awsvpc_configuration:
      subnets:
        - "<SUB_1_ID>"
        - "<SUB_2_ID>"
      security_groups:
        - "<SG_ID>"
      assign_public_ip: ENABLED
EOF

Next, create the ECS service from your compose file using the ecs-cli compose service up command. This command will look for your docker-compose.yml and ecs-params.yml files in the current directory. Replace the <CLUSTER_NAME> variable with the name of your ECS cluster and the <PROJECT_NAME> variable with the desired name of your ECS service.

ecs-cli compose \
  --project-name <PROJECT_NAME> \
  --cluster <CLUSTER_NAME> \
  service up \
  --launch-type FARGATE

Allow some time for your ECS service to deploy. You can now check the status of your ECS service using the ecs-cli ps command.

ecs-cli compose \
  --project-name <PROJECT_NAME> \
  --cluster <CLUSTER_NAME> \
  service ps

By navigating to the IP address specified on port 8080, you will be able to view the homepage of our project, confirming that your task was able to successfully pull the container image from the GHCR registry using your credentials for authentication.

Cleanup

Stop your ECS service using the ecs-cli compose service down command.

ecs-cli compose \
  --project-name <PROJECT_NAME> \
  --cluster <CLUSTER_NAME> \
  service down

Delete the AWS CloudFormation stack that was created by ecs-cli up and the associated resources using the ecs-cli down command:

ecs-cli down --cluster <CLUSTER_NAME>

Deploy a React App to GitHub Pages

2024-06-03T08:30:00+00:00

Here, I will show you how to deploy a React app to GitHub Pages using GitHub Actions. GitHub Pages is a static site hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub and makes it accessible as a website. GitHub Actions is a CI/CD service that allows you to automate your workflow on publishing your static site to GitHub Pages.

Create a React App

First, create a new React app using Create React App.

npx create-react-app react-app --template typescript
cd react-app

Your git repository should be initialized automatically.

Create a GitHub Repository

Working in the folder with the newly created app, create a new repository on GitHub.

gh repo create

Adjust repository settings

Go to the repository settings and enable GitHub Pages. Choose the source for Build and deployment — GitHub Actions.

Open the Actions/General menu, scroll down to the Workflow permissions section, and give Read and Write permissions to the workflow.

Add GitHub Actions Workflow

Create a new directory called .github/workflows in the root of your repository.

mkdir -p .github/workflows

Create a new file called deploy.yml in the .github/workflows directory.

Name your workflow.

name: Deploy React App

This will allow you to distinguish this workflow from others.

Add the following content to the deploy.yml file.

…

on:
    push:
        branches: [main]
    pull_request:
        types:
            - closed
        branches: [main]
    workflow_dispatch:
…

This section describes the events that will trigger the workflow. In this case, the workflow will be triggered on push to the main branch, when a pull request is closed, and when the workflow is manually triggered.

Next, add jobs description to the workflow.

…
jobs:
  build:
    runs-on: ubuntu-latest
…

Our job with the name build will run on the latest version of Ubuntu.

Add steps to the job.

…
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
…

The first step checks out the code from the repository. The second step sets up Node.js.

Add more steps to the job.

…
      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build
…

The third step installs dependencies, and the fourth step builds the React app.

Add the final steps to the job.

…
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          name: 'github-pages'
          path: build
…

Here we use the upload-pages-artifact action to upload the build directory as an artifact for further deployment.

Create deployment steps.

…
  deploy:
    if: github.event.pull_request.merged == true || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
    needs: build

    permissions:
      pages: write
      id-token: write
      contents: read

    environment:
      name: github-pages
      url: url: ${{ steps.deployment.outputs.page_url }}
…

The deployment job will run only if the pull request is merged or the workflow is triggered by a push event or fired manually. It needs the build job to be completed. The deployment job has permissions to write to the pages, write the id-token, and read the contents. The environment name is github-pages, and the URL is the output of the deployment step.

Add steps to the deployment job.

…
    runs-on: ubuntu-latest
    steps:
      - name: Setup Pages
        uses: actions/configure-pages@v5

      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          artifact_name: 'github-pages'

The first step sets up the Pages. The second step deploys the build directory to GitHub Pages using the deploy-pages action. We use the previously built artifact and the GitHub token for authentication of the deployment.

Commit the changes and push them to the repository.

git add .
git commit -m "Add GitHub Actions workflow"
git push

Adjust `package.json`

Open the package.json file and add homepage property. You can set it to . or provide the URL of your GitHub Pages ("homepage": "https://andygol.github.io/react-app/").

{
  "name": "react-app",
  "version": "0.1.0",
  "private": true,
+  "homepage": ".",
  "dependencies": {
    …

Deploy the React App

Any edits you make to the main branch will trigger the GitHub Actions workflow. The workflow will build the React app and deploy it to GitHub Pages.

https://andygol.co.ua/react-app/

That’s it! You have successfully deployed a React app to GitHub Pages using GitHub Actions.

PS. If you want to see the full workflow file, check out the GitHub repository.

OpenStreetMap ain’t Google Maps

2023-07-22T08:30:00+00:00

There should be a screenshot from Toy Story where Buzz tries to explain to Woody something about the stars by showing his hand up.

About 20+ years ago, I came across a short fiction story on a computer-related theme with elements of a thriller, in which there was a moment when the main character hastily created some geoservices, applications that use them, almost from nothing, to get out of situations in which he found himself. At that time, I did not fully understand the whole idea, because I had never encountered such a thing in my life. And here we are in the new century/millennium, we have a device in our pocket that uses these mysterious geoservices — a smartphone.

Let’s try to figure out these mysterious geoservices. In fact, they existed long before the appearance of smartphones, and even before the appearance of computers in general. The simplest example is postal addresses. Knowing the number of the house, the name of the street, you can get to the destination. You can plan (think over) the route on your own by drawing an approximate diagram on a piece of paper, or by making a similar one in your head. In fact, we do this every time we leave the house – we plan our route and then stick to that plan, even when we go to the nearest store to buy groceries. If you are going to an unfamiliar, remote place about which you either have little information, or you have never been there, you ask your friends how to get there, what type of transport to use, which stop to get off at, which direction to go next and where to turn. You can take a taxi and the driver will take you to the specified address or landmarks. If you need to send a parcel to another city, the postal service will provide you with such a service, but you still need to specify the recipient and the place where the parcel should be delivered. As you can see, we need information about the location of some object in the area (in space), that is, geospatial information. This information is the basis of all geoservices.

… then there will be a beet field, then a forest strip, then a corn field, then a wheat field, a hemp field, behind it a talking river — it will tell …

Geoservices are information and technological solutions that provide collection, processing, storage and provision of geographic (geospatial) information. They help to understand and use spatial data for various purposes such as navigation, data analysis, planning, marketing and many more. With the help of geoservices, users can interact with maps, exact coordinates, geographic objects and other geodata.

And now it is almost impossible to imagine calling a taxi, ordering a pizza delivery, and calculating the potential reach of the target audience without the use of geospatial data and services.

I have repeatedly seen people confuse data with services. This is exactly the case mentioned in the title. I have seen several different analyses where the authors tried to compare Google Maps exactly with OpenStreetMap without having any idea of what they were comparing. Let’s figure it out.

Let’s talk about Google Maps first

The history of Google Maps began in the mid-2000s and is an interesting example of a successful combination of innovative technology and practical application.

Google Maps was launched on February 8, 2005. The maps were created as a project of the Google team, which sought to create a map web service with high map loading speed and interactive capabilities.

The first version of Google Maps provided the ability to view maps, search for places, plan routes, and navigate using GPS. The interface was as intuitive and user-friendly as possible.

As early as June 2005, Google introduced the Google Maps API, which allowed developers to integrate mapping functions into their websites and applications. This opened wide opportunities for the use of cartography in other projects.

In 2007, Google Maps was supplemented with a new functionality — Street View, which allowed users to view photos along roads and streets, which gave them an idea of a realistic view of the area, there was an opportunity to virtually visit other places in which you have never been.

In 2008, a mobile version of Google Maps for smartphones was launched, allowing users to access maps and navigation directly on their mobile devices.

In 2010, Google integrated the functionality of Google Maps with another product — Google Earth, allowing users to view 3D models and images from satellites.

Google Maps continued to evolve, adding new features such as turn-by-turn navigation, offline mode, place ratings and reviews, event and establishment recommendations, and more.

Today, from the point of view of the average user, Google Maps has become one of the most popular tools for navigating and exploring the world. They are used by millions of people every day to find directions, find places, view photos and explore new remote locations. Google Maps has also become an important resource for businesses, researchers and public organizations that use its services to solve various tasks.

OpenStreetMap

The history of OpenStreetMap (OSM) begins in 2004 and is connected with the desire to create a free, open and accessible database of geospatial information.

OpenStreetMap was founded in Great Britain by Steve Coast in August 2004. He had the idea to create a global mapping platform where users could collaboratively create and edit geodata.

The main reason for the emergence of OpenStreetMap was the lack of available and up-to-date geographic information in some regions of the world. Commercial global mapping services such as Google Maps (which did not exist at the time) did not always provide coverage of sufficient quality or detail for certain areas.

OpenStreetMap put forward the principle of open data and access to cartographic information. The project became a platform for a global community of volunteers who could join the process of creating and editing geodata of any region of the planet.

Over time, OpenStreetMap attracted the attention of more and more users and built an active community. Volunteers began to actively enter data on roads, localities, information on hydrography, enterprises, institutions and other geographical objects.

OpenStreetMap’s development-oriented community has become one of the largest and most active free mapping communities in the world. Through open and free data access, openness and collective efforts, OSM has become a source of valuable and relevant geospatial information used in a variety of industries, including tourism, research, humanitarian aid and urban planning.

Goals and philosophy

Google Maps is a commercial product created by Google. The main goal of Google Maps is to provide users with convenient and fast interactive maps for navigation, finding places and orientation. Google makes money from it by displaying ads and providing paid services.

OpenStreetMap, on the other hand, is a project based on volunteer principles and an open data philosophy. The main goal of OSM is to create a free and publicly available geospatial database for the entire world. Everyone can contribute by adding new data or correcting issues. This approach makes it possible to create detailed and up-to-date maps where other sources may be limited or outdated.

Data sources

Google Maps uses commercial and proprietary data sources, licensed data from third-party companies, and machine learning algorithms to process large volumes of data accumulated by the company along with on-site data collection using specialized equipment and customer device tracking. This gives them an advantage in speed and coverage, but hides details and sources of information. (Thus, all of us moving around the area unknowingly help Google sell us services using data received from our smartphones in the form of automated reports from applications or the OS.)

OpenStreetMap uses open data sources, such as satellite images provided by suppliers for the purposes of the project, open geospatial data distributed freely by national governments, data from other free open sources, as well as own contributions of project participants. This allows for more transparent and verifiable data that can be useful to the public, researchers and even humanitarian organizations.

Data relevance

Google Maps has access to resources that allow updating their database with a certain periodicity, especially where there is an increased demand for them and more money can be made from the services. This helps provide users with up-to-date information such as traffic jams, entertainment and new establishments. (Remember your smartphones that Google collects data on all of this?)

OpenStreetMap, depending on the region, may have a different level of data relevance. Here everything depends on the number and activity of volunteers who make changes to the database. However, where there is an active community of project participants, the data can be quite relevant and detailed.

You can notice that both Google Maps and OpenStreetMap have similar elements on the main page:

Map
Search
Ability to build a route
View information about the object on the map…

So what’s the difference then?

The main difference is that Google Maps is a commercial product whose purpose is to provide services to customers. OpenStreetMap is primarily focused on data collection and distribution.

– And what about the API? - you can say.
– This is not a single API, but a collection of various APIs, each of which is responsible for providing one or another service.

API for displaying maps; API for search (geocoding); API for routing; APIs for routing and providing turn-by-turn instructions and everything else are all these components that make up the final product.

Google’s advantage is centralization, all APIs are developed and maintained in-house. Using them, you expect to receive a certain volume of services with a certain level of availability and quality (SLA), for which you are ready to pay a certain amount, without the need to deploy the entire infrastructure at your own or at the client’s. Calculating how much it will cost you to use the services is quite non-trivial, because sometimes it is quite difficult to understand how some services depend on others and to predict the expected level of costs.

Instead, the OpenStreetMap API is intended for receiving, editing and saving data in a common geospatial database by project participants. That is, the only thing that OpenStreetMap guarantees you is that you can use the data and already using this data you can create your own map, geocoder, offer your customers navigation services or something else. The map, search, routing on the Main page of the OpenStreetMap site is a demonstration of what can be done with project data, all these elements are separate independent projects in themselves and the only thing that connects them to OpenStreetMap is the data that OpenStreetMap provides to everyone.

Map — tile server (Mapnik) and map style (OSM-Carto) — 2 separate projects, which are in no way dependent on OSMF (OpenStreetMap Foundation)
Tile Display Library — leaflet.js, an open source JavaScript library for displaying interactive maps
Search — geocoder Nominatim, also an independent project
Routing — OSRM, Valhalla, GraphHopper, also third-party projects
Data export/mining — Overpass API (Overpass-Turbo), ditto…
iD is an OSM data editor (built into Main), now managed by OSMF.

All these are free projects with open source code and you, if necessary, if possible, if you have the appropriate qualifications, you can assemble the solution you need from these building blocks. Especially since you have a choice among various data rendering projects, map creation; various geocoding and search engines; routing and navigation engines. In addition, you can use OpenStreetMap data for analysis in your own projects, combining it with your other data, which is something you cannot do with Google Maps.

Using OpenStreetMap

OpenStreetMap is used by various companies and organizations, including Meta, Apple, Amazon, TomTom and others.

Meta uses OpenStreetMap data for its mapping and location integration features. To provide its users with up-to-date information about places and locations; Meta uses OSM data to show users’ locations, maps and integrate with other features on its platforms.

Apple also uses OpenStreetMap data in some of its services. For example, in some countries or regions where data from third-party providers for Apple Maps may be less complete or out of date, or conversely where OpenStreetMap data is of higher quality, they may use it to support navigation and cartography.

Amazon also uses OpenStreetMap data in its services such as AWS (Amazon Web Services). OSM can be used in a variety of geographic data processing, analysis, and visualization solutions on the AWS platform.

Among other manufacturers of navigation equipment, TomTom also cooperates with OpenStreetMap. They use OSM data to provide navigation and mapping services in their devices and applications.

Airbnb, a popular accommodation booking platform, also uses OSM data to display locations and accommodations on its maps. They can use OSM to provide accurate information about the location of residences and their surroundings.

ESRI ArcGIS Online allows users to integrate OpenStreetMap data into their geographic information systems. Users can import OSM data into their projects and use it for analysis, mapping, and visualization.

These examples show a wide range of uses of OpenStreetMap by well-known companies and services. The openness and availability of OSM data allow different platforms and organizations to use geographic information to provide quality services and develop their own products.

It should be noted that the choice of what to use, Google services, or OpenStreetMap data and ecosystem, depends on specific requirements, the completeness and relevance of data coverage in a particular area, the need for quickly received changes in data, the possibility of implementing data quality control processes and availability you have experience working with Google products or the OpenStreetMap ecosystem and data. The choice is yours.

OSM 2.0 API using git

2023-05-07T08:29:00+00:00

The OSM 2.0 API (OpenStreetMap) is an application programming interface that provides access to geospatial data and its change history stored with git. The basic data structure in OSM should be based on dividing the globe into tiles, each of which is a separate git repository.

The OSM object data storage format is yaml (YAML Ain’t Markup Language). Yaml is an easy-to-read and easy-to-save data format based on the use of indentation to represent data structure. Using yaml in OSM allows storing various multi-level attributes in the form of tags for geographic objects.

Each object will be stored as a separate YAML file, unlike previous versions of the OSM API where data was stored and exchanged as XML files.

Each tile in OSM is an independent git repository. Git is a version control system that allows you to track the history of changes in a file system and store different versions of files. Using git for OSM allows you to track all changes made to geospatial data and store their history.

Description

The OSM API should provide various methods for accessing and modifying data. Some of these should include:

Obtaining data by coordinates or area.
Search for objects by type, attributes or tags.
Obtaining the history of changes for a specific object.
Addition of new objects or changes to existing ones.
Deleting objects.

The OSM API should provide the ability to perform queries based on various protocols to conveniently retrieve and interact with geospatial data.

Overall, the OSM API, which uses git to store geospatial data and its change history, provides a powerful mechanism for collaborating and managing geospatial data. Key features of the OSM API using git include:

Version control: By using git, the OSM API allows you to store a change history for each object. This means that you can track who made changes to a specific object, when, and what changes, as well as restore previous versions of the data.

Distributed data structure: The globe is broken down into tiles, which are self-contained git repositories. This allows you to effectively manage and distribute the load on different servers. Each tile contains only the objects within its boundaries, which makes it easier to access specific data.

Data format flexibility: Using the yaml format to store object data allows you to store various attributes and tags that help describe geographic objects in detail. This allows you to add your own tags and attributes for more flexible data presentation.

Scalability: The distributed data structure and the use of git make it easy to scale the system to handle large volumes of geospatial data. You can add new tiles or servers to expand the infrastructure and improve performance, break large tiles into smaller ones over time.

Convenient access to data: The OSM API provides a variety of methods for accessing OSM data. You can retrieve data by coordinates or regions, search for objects by type, attributes, or OSM tags, and retrieve change history for a specific object. This provides flexibility and the ability to select only the data you need.

Possibility of joint work: Using git allows multiple users to work with data simultaneously and make changes. Thanks to version control and the ability to merge branches, you can easily work together on projects, make and merge changes.

Integration with other tools: Because OSM uses git, you can use a wide variety of tools that support git to work with geospatial data. These can be various version control systems, code editors, data analysis tools, and others.

In addition to these capabilities, the OSM API using git allows you to:

Data extension: You can add your own data to OSM data, extending the current geospatial database. It allows you to create your own datasets, add attributes, and extend the functionality of OSM.

Integration with other services: The OSM API can be easily integrated with other services and tools, such as geospatial services or data analysis systems. This allows you to combine different data sources and use them to create the right solutions.

Advanced editing options: The OSM API allows you to make changes to geospatial data, including adding new features, editing attributes, and deleting features. This makes it possible to actively cooperate with the global OSM database and supplement it with knowledge about specific objects or regions.

These are just a few additional features of the OSM API using git. Overall, the OSM API should be a powerful tool for working with geospatial data, providing version control, data format flexibility, and scalability.

YAML is for saving data

OSM API version 2.0 uses the yaml format. Each object is stored as a separate YAML file, and this differs from previous versions of the OSM API, where data was stored as XML files. Using the YAML format to store individual object files has several advantages:

Ease of reading and editing: The YAML format has a clear, easy-to-read syntax that makes it easy to manually edit or view data. Using indentation instead of XML syntax elements makes it easier to work with data and edit it

Flexibility in data expressiveness: YAML allows you to store additional properties and information about objects that are necessary for a particular application. It can include simple values, lists, associative arrays, and nested data structures.

Data search and filtering: Being individual files, objects can be easily found and filtered by various parameters. You can use YAML libraries or tools to search and manipulate data as needed.

Change history support: Each object is presented in the form of its own file, which allows you to save the history of changes and track the development of data. Git, as a version control system, can be used to manage changes and branches, which facilitates effective collaboration and data version control.

Data expansion and validation: You can use YAML schemas to validate and control the correctness of data. This ensures that the data conforms to the given rules and structure. Also, you can extend YAML schemas to define additional properties and constraints for stored objects.

Integration with other tools: YAML is a popular data exchange format supported by many tools and platforms. You can use various tools to import and export data, and to process and analyze geospatial data in YAML format.

Git repository as a data store

The shape (geometry) of the tiles of each repository can be arbitrary.

In the OSM API using git, the shape (geometry) of the tiles of each repository can be arbitrary. Tiles can be of different shapes and sizes, depending on your geospatial data needs.

A description of tile boundaries, or other metadata that pertains to each tile, can be stored in the tile repository itself. This allows important information to be stored alongside geospatial data, helping to ensure consistency and availability of that data.

In addition, there is a master repository that includes tile repositories in the form of git submodules. The master repository serves as the starting point for managing and coordinating tile repositories. It may contain additional information about the tiles, such as indexes or links to each tile.

This architecture allows geospatial data to be organized and managed by storing the description of tile boundaries in the tile repository itself and using a master repository to coordinate and access these tile repositories.

This approach provides flexibility and scalability when working with geospatial data, allowing easy management of individual tiles, as well as maintaining change history and version control using git.

Data segmentation into tile repositories allows you to implement:

Support for distributed cooperation: By using git, the OSM API enables distributed collaboration. Different users can work on different tiles, make changes and interact with the OSM database using the same principles as for working with code, and later merge their changes using git.

Synchronize and merge changes: By using git, the OSM API makes it easy to synchronize and merge changes made by different users into their own copies of repositories. This avoids conflicts and ensures data integrity when merging changes. In addition, users can make their own forks to add their (mostly non-public) data on top of OSM data. Over time, when the decision is made to push your own closed data into the public domain, this can be done by using git push. Changes can be pushed both upstream, from local copies to shared repositories, and downstream, from shared to local repositories, allowing users to keep their own repositories up-to-date while receiving updates from the community, with less effort to synchronize data.

History of changes: Thanks to the use of git, the OSM API stores the history of object changes in each tile, which allows you to follow the development and changes of geospatial data. This is a useful feature for analyzing data, tracking changes, and restoring previous versions.

Version support: Git allows you to store each version of an object (file) in a separate commit, which makes it possible to easily review, restore, and track data changes over time. You can view the change history and go back to previous versions if needed.

Easy replication and distribution: Because each object is stored in its own file, they can be easily replicated and distributed across servers or tiles. This allows for improved scalability and data access. Git provides the ability to synchronize and replicate data between different servers or tiles. You can use git features, such as branching and merging, to manage changes and updates to your data.

Conflict management: In the case of simultaneous changes to the same object, git provides a means to resolve conflicting changes and merge different versions. This allows multiple users to work with the same data and handle conflicts efficiently.

Data recovery: Thanks to git version control, the OSM API allows you to restore data to previous versions or restore deleted objects. This ensures data safety and security.

OSM changeset – git commit

So, in the context of saving OSM data in git, a changeset is represented as a commit in git. A commit is a fixed data structure that captures the changes made at a specific point in time.

Each commit in git contains information about the changes made to the repository. In the case of OSM, a commit can contain changes related to the addition, deletion, or modification of geospatial objects.

Git commits provide a history of changes and allow you to track the development of data over time. Each commit has a unique identifier (hash), which serves to identify them and restore a certain version of the data.

One commit can contain changes for one or more geospatial objects. This allows you to record changes in individual objects and manage the history of changes independently of other objects.

Commits can be grouped into branches and merged with each other, which allows you to manage parallel branches and merge their changes into one common version of the data.

Hence, commits in git are used to save and manage changes to OSM geospatial data in the git version of data storage.

Advantages of using git

git commands: You can use standard git commands to manage tile repositories, such as git clone, git pull, git push and others. It allows you to work with the geospatial data of the OSM API using the familiar and powerful git interface.

Advanced git features: The OSM API can use additional features and capabilities of git, such as branches, git-tags, and commits. It allows you to organize and manage different versions, create milestones and work with different branches.

Access control: The OSM API can provide the ability to restrict access to individual tiles or data using git’s access control features. This allows you to control who has the right to read, write or do other things with geospatial data.

Backup and Restore: Thanks to git, the OSM API allows you to back up your data and restore it when needed. This provides an additional layer of protection and security for your geospatial data without having to develop your own solutions for it.

Distribution of changes: You can easily propagate changes made to tile repositories to other OSM API instances or installations using familiar git methods. No need to waste server resources creating data dumps and data for replication. With the use of git, this is done by fetching updated data as needed with the git pull command. It allows collaboration with different communities or distributed systems, sharing and synchronizing geospatial data.

Integration with other tools: The OSM API using git can be easily integrated with other development tools, such as project management systems, automation tools, cloud storage services, and others. This makes it an important component when developing geospatial applications and working with data.

Synchronization with other data sources: Using the OSM and git APIs, you can synchronize geospatial data with other data sources. This can be useful if you have additional data sources or want to combine data from different sources to create a more complete and comprehensive database.

Extensions and customizations: The OSM API with git allows you to extend and customize its functionality according to your own needs. You can add additional modules, functions, and extensions to work with geodata and interact with the API.

Using the OSM API with git provides powerful capabilities for managing geospatial data, maintaining its change history, and collaborating with the community.

Summary of OSM API capabilities using git

The OSM API, which uses git to store geospatial data and change history, has the following features:

Each tile is an independent git repository containing a separate set of geospatial data.
Tiles can have a different shape (geometry) and the description of their boundaries is stored in the tile repository itself and the master repository.
Data about individual objects is stored in the YAML format, which allows you to store additional properties and information about them, imitating all the flexibility of the OSM tagging system.
The API supports distributed collaboration, allowing different users to work on different tiles and merge changes using git.
Changes are saved in the change history of git repositories, which allows you to follow the development of geospatial data.
Standard git commands like clone, pull and push can be used to manage tile repositories.
The API provides advanced git features such as branches, git tags, and commits for version control and development.
There is an option to restrict access to tiles and data using git access control.
The API allows you to restore data to previous versions and perform data backup and recovery.
Changes can be propagated to other instances of the OSM API and collaborated with the OpenStreetMap community and other developers.

Distribution of large objects between tiles

The distribution of large objects between tiles is an important aspect when organizing geospatial data in the OSM API using git. This allows you to effectively manage large volumes of data and ensure optimal performance and speed of access to this data.

There are several approaches to distributing large objects between tiles in the OSM API. Here are some of them:

Geographic distribution: Objects can be distributed between tiles using their geographic location. For example, large regional objects can be divided into separate tiles covering the corresponding territory. This allows data to be stored closer to its physical location and reduces the load on individual tiles.

Distribution by categories: Objects can be distributed between tiles depending on their categories or types. For example, road or hydrographic features can be stored in separate tiles, making it easier to access specific categories of data.

Dependence on size: Objects can be divided into tiles depending on their size or complexity. For example, large or complex objects can be stored in separate tiles for optimal management and quick access to them. It can be contours of oceans and continents, borders of countries and so on.

Dynamic allocation: The OSM API can automatically allocate large objects between tiles based on load and demand.

An alternative approach may be to distribute large objects based on virtual tile boundaries. Instead of physically distributing objects between separate tile repositories, you can use virtual borders that join objects that are logically related to each other.

In this approach, large objects can be divided into multiple tiles, and these tiles can share a repository or have references to each other. This allows objects that are physically located in different tiles to be stored together in a virtual group.

Accordingly, you can have each tile as a separate git repository, but they can reference each other to form a virtual group of objects. This provides convenient access and management of large objects located in different tiles, while maintaining the flexibility and speed of git.

The choice of approach to the distribution of large objects between tiles depends on the specific needs and requirements for performance, availability, and manageability of the data.

Notes

It is important to note that this description is general and specific details and implementation may depend on your specific OSM 2.0 API implementation and your needs.

When working with the OSM API using git, make sure you follow OSM’s rules and policies for making changes to the global database. Keeping a change history and using git correctly will help manage these changes and ensure data security.

If you have specific questions about using the OSM API with git or need more details, please let me know and I’ll try to give you more details.

Andrii Holovin – Blog |

Deploying an HA Kubernetes cluster with an external etcd topology on a local machine

Cluster Architecture

Components

Cluster Topology

Prerequisites

SSH Key Setup

Generating an SSH Key

Updating cloud-init Configuration

Accessing Virtual Machines via SSH

Environment Preparation

Creating Scripts and Configurations

Virtual Machine Parameters

Cloud-init Configurations

User Creation

Base Configuration for Cluster Nodes

Configuration for etcd nodes

Configuring the HAProxy balancer node

Choosing a network topology

Creating an etcd cluster

Deploying the first node of the etcd cluster

Configuring the first etcd node

Deploying subsequent etcd cluster nodes

Configuring etcd nodes and joining them to the cluster

Creating kubeadmcfg.yaml

Canceling node joining to the cluster

Removing data-dir

Configuring the load balancer for control plane nodes (HAProxy+Keepalived)

Temporary “hack” (if you can’t set up HAProxy right now)

Configuring HAProxy+Keepalived

Netplan configuration

Keepalived configuration

Kernel configuration (Sysctl)

How it works together

Deploying HAProxy+Keepalived

What to do after launch?

Deploying the Control Plane

Configuring a local load balancer for access to etcd nodes

Preparing the HAProxy configuration

Creating a Static Pod for HAProxy

Deploying the first control panel node

Checking the use of Static Pods HAProxy

Configuring kubeadm

Starting initialization and bypassing ExternalEtcdVersion verification

Installing Calico

What to do first: deploy Calico or run kubeadm join?

Option 1: CNI first, then Join (Recommended)

Option 2: Join first, then CNI

Deploying and joining the subsequent control panel nodes

Disconnecting and replacing a control panel node

Deploying Worker Nodes

Installing MetalLB

Testing the cluster

Summary

Appendices

Network architecture for HA Kubernetes with external etcd

Recommended network topology — Subnet: 10.10.0.0/22 ​​(1024 addresses)

1. Infrastructure (10.10.0.0/26)

2. etcd Cluster (10.10.0.64/26)

3. Control Plane (10.10.0.128/25)

4. Worker Nodes (10.10.1.0/24)

Kubernetes internal networks (do not conflict with VMs)

Alternative subnet options

Option 1: Compact topology (10.10.10.0/24)

Option 2: Extended topology (10.10.0.0/16)

Option 3: Classic private network (172.16.0.0/16)

DNS records (recommended)

Troubleshooting

Fix --pod-network-cidr

1. Update the ClusterConfiguration

2. Update Control Plane Manifests (on all 3 nodes)

3. Update kube-proxy settings

4. Reinstall Flannel with the correct CIDR

5. Update Node Specs (PodCIDR)

6. Final Reboot

Resetting etcd to its original state

1. Stop services that depend on etcd

2. Clear data on all nodes

3. Initialize a new “empty” cluster

4. Cleanup via etcdctl (if cluster is running)

Recommended network topology — Subnet: 10.10.0.0/22 (1024 addresses)

Fix `--pod-network-cidr`