Orthevs Orthevs Fathom new horizons
Μενού

Wiki workshop · Αγγλικά

Provisioning: Kubernetes on Bare Metal

This document describes the complete provisioning of Kubernetes in bare-metal environments. It guides you from hardware preparation through operating system setup, container runtime, cluster bootstrap, networking, load balancing, storage…

Γλώσσα περιεχομένου · ΑγγλικάΜέγεθος · 99.2 KBdocs/environment-native-baremetal-kubernetes-EN.md
Παραδείγματα workshop σε ZIP
Πρόοδος μάθησης στη web app

Σημειώστε κεφάλαια ως αναγνωσμένα, δείτε την πρόοδο ανάγνωσης και ολοκληρώστε πρακτικές ενότητες αφού ενεργοποιήσετε την προαιρετική web app. Η πρόοδος του workshop υπολογίζεται από τα κεφάλαια όλων των σελίδων τεκμηρίωσης. Οι επιλογές μένουν σε αυτή τη συσκευή· δεν υπάρχει λογαριασμός ή συγχρονισμός με server. Δεν απαιτείται εγκατάσταση. Η ανάκληση στις Ρυθμίσεις απορρήτου διαγράφει την πρόοδο.

Πρόοδος ανάγνωσης0%
0%
0 / 39 κεφάλαια που διαβάστηκαν

Δεν υπάρχει ακόμη ελληνική έκδοση αυτού του εγγράφου. Το περιεχόμενο εμφανίζεται στα αγγλικά.

Άλλη γλωσσική έκδοσηΓερμανικά ↗

Provisioning: Kubernetes on Bare Metal

This document describes the complete provisioning of Kubernetes in bare-metal environments. It guides you from hardware preparation through operating system setup, container runtime, cluster bootstrap, networking, load balancing, storage, add-ons, validation, troubleshooting, upgrades, and teardown.

The file is intentionally practical. You can use it in a workshop as a guide, as a checklist, or as the basis for your own automation with Bash, Ansible, PXE, cloud-init, Talos, k3s, RKE2, or k0s.

Important: Bare metal is not a single setup. You make architecture decisions about hardware, firmware, networking, storage, load balancers, operating system, Kubernetes distribution, CNI, CSI, Ingress, certificates, backup, and the operating model. This document makes those decisions explicit.

1. Purpose

This file documents how to provision Kubernetes reproducibly on physical hardware. It does not replace a project-specific operations manual, but it provides a solid reference for planning, provisioning, and validation.

Use this document when you want to:

  • build a bare-metal workshop cluster,
  • prepare multiple physical servers in a lab or data center,
  • create a production-like kubeadm cluster with control-plane high availability,
  • evaluate an edge environment with k3s, RKE2, k0s, or Talos,
  • provide LoadBalancer services without a cloud provider,
  • choose storage and networking consciously for bare metal,
  • prepare air-gapped or restricted environments,
  • define a troubleshooting and teardown process.

1.1. Target State

At the end, you should have more than a running cluster. You should have a traceable platform foundation:

flowchart TD
    Hardware[Physical Servers] --> Firmware[Firmware, BIOS, RAID, BMC]
    Firmware --> OS[Operating System or Kubernetes OS]
    OS --> Runtime[Container Runtime]
    Runtime --> Bootstrap[Cluster Bootstrap]
    Bootstrap --> Network[CNI, Service Network, DNS]
    Network --> LB[Bare-Metal LoadBalancer]
    LB --> Storage[Storage and CSI]
    Storage --> Addons[Ingress, Gateway, Certs, Observability]
    Addons --> Validate[Validation]
    Validate --> Operate[Operations, Upgrade, Backup]

1.2. Scope Boundaries

This document describes provisioning the Kubernetes platform. It does not cover the following in depth:

  • application architecture for individual workloads,
  • deep SOC-grade security monitoring,
  • full network-router configuration for every vendor device,
  • full storage-array configuration for every vendor,
  • production compliance documentation.

These topics are covered only to the extent required for bare-metal provisioning.

01 / 39

2. Core Decisions

Bare-metal Kubernetes depends more strongly on the environment, network, and storage than a managed Kubernetes service. Define the target architecture before installation.

2.1. Decision Matrix

Scenario Suitable approach Typical size Advantage Disadvantage
Local workshop Single-node kubeadm, k3s, or k0s 1 node Fast, transparent, inexpensive No real HA
Multi-node lab kubeadm with one control plane and multiple workers 2 to 5 nodes Close to upstream Kubernetes, easy to explain Control plane is a single point of failure
Production-like lab kubeadm HA with stacked etcd 3 control-plane nodes + workers Close to common standards, good learning value Load balancer required
Data center kubeadm HA with external etcd or RKE2 3+3+N nodes Clear roles, robust More operational effort
Edge / small hardware k3s 1 to 5 nodes Low resource footprint Distribution abstracts details
Security-focused RKE2 or Talos 3+ nodes Hardened operating models Different operating concepts
Immutable OS Talos Linux 1 to N nodes Declarative machine configuration No SSH, different troubleshooting
Full automation PXE, MAAS, Matchbox, Metal3, Ansible N nodes Reproducible, scalable Higher initial complexity
Air-gap kubeadm, RKE2, k3s, or Talos with local registry depends Controlled, offline-capable Artifact management is critical

2.2. Guiding Rule for Workshops

For a workshop, kubeadm is didactically valuable because you install the Kubernetes building blocks visibly: operating system, runtime, kubelet, kubeadm, kubectl, CNI, control plane, worker join, and add-ons.

For edge-like and appliance-like environments, k3s, RKE2, k0s, or Talos are often more appropriate because they package cluster lifecycle and host operations more tightly.

2.3. Decision Diagram

flowchart TD
    Start[You need Kubernetes on bare metal] --> NeedHA{Do you need control-plane HA?}
    NeedHA -- No --> Small{Small edge or lab hardware?}
    Small -- Yes --> K3s[k3s or k0s]
    Small -- No --> SingleKubeadm[Single-node or multi-node kubeadm]
    NeedHA -- Yes --> Ops{Do you want to learn Kubernetes components close to upstream?}
    Ops -- Yes --> HAkubeadm[kubeadm HA]
    Ops -- No --> Dist{Do you need a hardened distribution?}
    Dist -- Yes --> RKE2Talos[RKE2 or Talos]
    Dist -- No --> K0s[k0s or Kubespray]
    HAkubeadm --> LB{Do you have an external load balancer?}
    LB -- Yes --> ExternalLB[HAProxy, F5, A10, NetScaler, NGINX, Router]
    LB -- No --> KubeVIP[kube-vip or HAProxy plus Keepalived]
02 / 39

3. Target Environments

This document distinguishes several bare-metal environments. You can combine individual sections, but you should not mix multiple cluster-bootstrap methods without a clear plan.

3.1. Native Linux Servers with kubeadm

This is the most transparent path:

  • Ubuntu, Debian, RHEL, Rocky Linux, AlmaLinux, Fedora Server, SLES, or a similar distribution,
  • containerd or CRI-O as the runtime,
  • kubeadm for bootstrap,
  • CNI of your choice,
  • MetalLB or kube-vip for LoadBalancer services,
  • CSI storage based on your infrastructure.

This path is especially useful for learning environments, platform teams, and production-like workshops.

3.2. General-Purpose Linux with a Kubernetes Distribution

Here you use a Kubernetes distribution that bundles installation and components more strongly:

  • k3s: lightweight, good for edge, small labs, ARM, and compact clusters.
  • RKE2: more focused on security and enterprise operations.
  • k0s: lightweight CNCF-conformant distribution with a simple controller/worker model.
  • Kubespray: Ansible-based Kubernetes deployment on existing machines.

3.3. Kubernetes-Optimized Operating System

Here the operating system itself is designed for Kubernetes operations:

  • Talos Linux: API-driven, no SSH, no traditional package management, declarative machine configuration.
  • Flatcar Container Linux: immutable host model, often with Ignition/Butane and automated updates.

These environments are strong operationally, but they require a different debugging model.

3.4. Fully Automated Bare-Metal Provisioning

If you provision many servers, you do not want to install manually from ISO. Typical building blocks are:

  • BMC/IPMI/Redfish for power and boot control,
  • DHCP, TFTP, HTTP, and iPXE,
  • Ubuntu Autoinstall, Debian Preseed, Kickstart, or Ignition,
  • MAAS, Foreman, Digital Rebar, Matchbox, Cobbler, or Metal3,
  • Ansible or Terraform as orchestration,
  • GitOps for cluster add-ons.
03 / 39

4. Reference Architectures

4.1. Single-Node Lab

A single node hosts both the control plane and workloads.

flowchart TD
    Dev[Your Client] --> API[kube-apiserver on Node 1]
    API --> ETCD[etcd on Node 1]
    API --> Kubelet[kubelet on Node 1]
    Kubelet --> Runtime[containerd or CRI-O]
    Runtime --> Pods[Workshop Pods]
    Pods --> CNI[CNI]
    Pods --> Storage[local-path, hostPath, or local PV]

Suitable for:

  • Kubernetes fundamentals,
  • YAML, kubectl, Deployments, Services,
  • simple CNI and storage demos,
  • local hardware tests.

Not suitable for:

  • real high availability,
  • production operations,
  • reliable upgrade tests with node drain,
  • tenant isolation.

4.2. Multi-Node Lab with One Control Plane

One control-plane node and multiple worker nodes.

flowchart LR
    Client[kubectl] --> CP[Control Plane Node]
    CP --> W1[Worker Node 1]
    CP --> W2[Worker Node 2]
    CP --> W3[Worker Node 3]
    W1 --> Pods1[Pods]
    W2 --> Pods2[Pods]
    W3 --> Pods3[Pods]

Suitable for:

  • scheduling,
  • DaemonSets,
  • CNI across multiple nodes,
  • Services and EndpointSlices,
  • rolling updates,
  • simulating worker-node failure without control-plane failure.

Risk:

  • The single control-plane node remains a single point of failure.

4.3. HA with Stacked etcd

With stacked etcd, etcd runs on the control-plane nodes.

flowchart TD
    Client[kubectl / API Clients] --> VIP[Control-Plane Endpoint]
    VIP --> CP1[Control Plane 1 + etcd]
    VIP --> CP2[Control Plane 2 + etcd]
    VIP --> CP3[Control Plane 3 + etcd]
    CP1 <--> CP2
    CP2 <--> CP3
    CP1 <--> CP3
    CP1 --> W1[Worker 1]
    CP2 --> W2[Worker 2]
    CP3 --> W3[Worker 3]

Recommendation:

  • at least 3 control-plane nodes,
  • an odd number of etcd members,
  • a dedicated, stable control-plane endpoint,
  • clean time synchronization,
  • fast and reliable network connectivity between control-plane nodes,
  • regular etcd backups and restore tests.

4.4. HA with External etcd

With external etcd, the etcd nodes are separated from the control-plane nodes.

flowchart TD
    Client[kubectl] --> LB[API Load Balancer]
    LB --> CP1[Control Plane 1]
    LB --> CP2[Control Plane 2]
    LB --> CP3[Control Plane 3]
    CP1 --> E1[etcd 1]
    CP1 --> E2[etcd 2]
    CP1 --> E3[etcd 3]
    CP2 --> E1
    CP2 --> E2
    CP2 --> E3
    CP3 --> E1
    CP3 --> E2
    CP3 --> E3
    CP1 --> Workers[Worker Nodes]

Advantage:

  • clear separation of API/controller/scheduler and etcd,
  • better control over etcd resources,
  • cleaner model for large production environments.

Disadvantage:

  • more machines,
  • more certificate and backup work,
  • more network and operational dependencies.

4.5. Edge Cluster with k3s

flowchart TD
    Server1[k3s Server 1] <--> Server2[k3s Server 2]
    Server2 <--> Server3[k3s Server 3]
    Server1 <--> Server3
    Server1 --> Agent1[k3s Agent 1]
    Server2 --> Agent2[k3s Agent 2]
    Server3 --> Agent3[k3s Agent 3]
    Client[kubectl] --> Fixed[Stable Server Address or LB]
    Fixed --> Server1
    Fixed --> Server2
    Fixed --> Server3

k3s is suitable for small sites, edge use cases, homelabs, and hardware with limited resources. You must still understand which components k3s already includes and which ones you want to replace.

4.6. Talos Cluster

flowchart TD
    Admin[talosctl on your client] --> TalosAPI[Talos API]
    TalosAPI --> CP1[Talos Control Plane]
    TalosAPI --> CP2[Talos Control Plane]
    TalosAPI --> CP3[Talos Control Plane]
    CP1 --> K8s[Kubernetes]
    CP2 --> K8s
    CP3 --> K8s
    K8s --> Workers[Talos Workers]
    Admin --> Kubeconfig[kubectl kubeconfig]
    Kubeconfig --> K8s

Talos differs from classic Linux hosts:

  • no SSH,
  • no manual package installation,
  • declarative machine configuration,
  • operations through talosctl,
  • strongly reduced host footprint.
04 / 39

5. Planning Checklist

Before you install, answer the following questions in writing.

5.1. Cluster Identity

Question Example
What is the cluster called? bm-lab-01
Which Kubernetes version do you use? v1.36.x or a project-wide pinned minor version
Which bootstrap method do you use? kubeadm, k3s, RKE2, k0s, Talos
What type of environment is it? Workshop, lab, edge, production, air-gap
Where is the desired state stored? Git repository, Ansible, Terraform, documentation
How is teardown performed? documented reset and data-deletion procedure

5.2. Node Plan

Role Minimum count Recommendation Notes
Control plane without HA 1 1 lab only
Control plane HA 3 3 or 5 odd etcd count
External etcd 3 3 or 5 dedicated fast disks
Worker 1 3+ depends on workloads
Load balancer 1 2 HAProxy/Keepalived, hardware LB, or kube-vip
Storage nodes 1 3+ relevant for Ceph/Longhorn/OpenEBS

5.3. Naming Scheme

Use clear hostnames. Example:

Node Role IP Notes
k8s-cp-01 Control Plane 10.10.10.11 first bootstrap node
k8s-cp-02 Control Plane 10.10.10.12 HA join
k8s-cp-03 Control Plane 10.10.10.13 HA join
k8s-w-01 Worker 10.10.10.21 workloads
k8s-w-02 Worker 10.10.10.22 workloads
k8s-lb-01 Load Balancer 10.10.10.5 optional
k8s-api.lab.local API endpoint 10.10.10.10 VIP or LB

5.4. CIDR Plan

Define the networks before kubeadm init. Changing them later is difficult.

Network Example Purpose
Management network 10.10.10.0/24 SSH, BMC, admin access
Node network 10.10.20.0/24 Kubernetes node IP, kubelet, API
Pod network 10.244.0.0/16 or 10.42.0.0/16 Pod IP addresses
Service network 10.96.0.0/12 ClusterIP Services
LoadBalancer pool 10.10.20.200-10.10.20.240 MetalLB or kube-vip
Storage network 10.10.30.0/24 Ceph, iSCSI, NFS, replication
BMC network 10.10.99.0/24 IPMI, iDRAC, iLO, Redfish

Ensure that pod and service CIDRs do not overlap with corporate networks, VPNs, or local networks.

5.5. Port Plan

Connection Port Direction Purpose
Client to API TCP 6443 Client → Control Plane Kubernetes API
kubelet API TCP 10250 Control Plane → Nodes Logs, exec, metrics
etcd client TCP 2379 API → etcd etcd access
etcd peer TCP 2380 etcd ↔ etcd etcd replication
kube-scheduler TCP 10259 local/Control Plane HTTPS
kube-controller-manager TCP 10257 local/Control Plane HTTPS
NodePort TCP/UDP 30000-32767 external → Nodes NodePort Services
CNI-specific variable Nodes ↔ Nodes VXLAN, Geneve, BGP, WireGuard
MetalLB L2 TCP/UDP 7946 Nodes ↔ Nodes memberlist when using L2 mode
BGP TCP 179 Nodes ↔ Router MetalLB/Cilium BGP
SSH TCP 22 Admin → Nodes classic Linux hosts
BMC variable Admin → BMC out-of-band management
05 / 39

6. Hardware and Firmware Preparation

Bare-metal provisioning starts before the operating system.

6.1. Minimum Hardware Values

Role CPU RAM Disk Network
Single-node lab 2 vCPU/cores 4 GB 40 GB SSD 1 Gbit/s
Control plane 2+ cores 4 to 8 GB 60+ GB SSD 1 Gbit/s, preferably 10 Gbit/s
Small worker 2+ cores 4+ GB 80+ GB SSD 1 Gbit/s
Production-like worker 4+ cores 16+ GB 200+ GB SSD/NVMe 10 Gbit/s
Storage node 4+ cores 16+ GB dedicated data disks 10 Gbit/s recommended
Dedicated etcd 2+ cores 8+ GB fast, low-latency SSD stable network

6.2. Firmware Checklist

Check on every server:

  • current firmware for BIOS/UEFI, NIC, RAID/HBA, BMC,
  • unique serial numbers and UUIDs,
  • virtualization enabled only if required for runtime features or nested labs,
  • correct boot order,
  • UEFI preferred; legacy boot only if needed,
  • Secure Boot enabled only if your distribution and runtime handle it cleanly,
  • NTP/RTC correct,
  • BMC/IPMI/Redfish reachable,
  • RAID/HBA mode aligned with the storage design,
  • no hidden vendor partitions with critical automation.

6.3. Disk Layout

For lab clusters, a single SSD is often enough. For production-like bare-metal clusters, separate the following more deliberately:

Mount Purpose Recommendation
/ operating system SSD, not too small
/var/lib/containerd images and container data dedicated partition or LV
/var/lib/kubelet pod data, volume mounts dedicated partition or LV
/var/lib/etcd etcd data fast dedicated SSD on control-plane nodes
/var/log logs separate limit or rotation
Storage disks PV/CSI/Ceph/Longhorn/OpenEBS do not mix with OS disk

6.4. BIOS and Power Profile

Set a consistent power profile:

  • Performance or Balanced profile instead of aggressive power saving,
  • restrict CPU C-states only when latency is critical,
  • evaluate NUMA and hyperthreading deliberately,
  • change NIC offloading only when you measure concrete network problems,
  • plan MTU consistently across switch, host, CNI, and overlay.
06 / 39

7. Network Design

Kubernetes on bare metal often fails not because of Kubernetes itself, but because of unclear Layer 2 / Layer 3 assumptions.

7.1. Network Model

flowchart TD
    Admin[Admin Network] --> API[API Endpoint 10.10.20.10:6443]
    API --> CP[Control-Plane Nodes]
    CP --> NodeNet[Node Network 10.10.20.0/24]
    NodeNet --> W1[Worker 1]
    NodeNet --> W2[Worker 2]
    NodeNet --> W3[Worker 3]
    W1 <--> PodNet[Pod Network 10.244.0.0/16]
    W2 <--> PodNet
    W3 <--> PodNet
    Users[Users / Clients] --> LBPool[LoadBalancer Pool 10.10.20.200-240]
    LBPool --> Services[Kubernetes Services]
    StorageNet[Storage Network 10.10.30.0/24] --> W1
    StorageNet --> W2
    StorageNet --> W3

7.2. Ground Rules

  • Every node needs a stable node IP.
  • Every node must be able to reach every other node.
  • Control-plane nodes must be reachable consistently via the API endpoint.
  • Pod CIDR and service CIDR must not overlap with existing networks.
  • CNI overlay ports must be allowed between nodes.
  • DNS and NTP must be reliable.
  • Firewall rules must not block kubelet, CNI, etcd, or the API.
  • MTU must match the CNI mode.

7.3. Static IPs or DHCP

Both are possible. For bare-metal Kubernetes, stable addressing is decisive.

Method Suitable for Recommendation
Static OS IP data center, small labs easy to trace
DHCP reservation PXE, MAAS, large labs easy to automate
Dynamic DHCP without reservation tests not for stable clusters
BGP-announced VIP production-like good with MetalLB/Cilium/kube-vip
VRRP VIP small HA clusters simple, but Layer-2-dependent

7.4. Multiple Network Interfaces

If a host has multiple NICs or default routes, control the node IP explicitly.

Example for kubeadm:

yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.20.11
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: 10.10.20.11
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
controlPlaneEndpoint: k8s-api.lab.local:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

7.5. DNS Plan

At minimum, these names should resolve:

Name Target
k8s-api.lab.local API VIP or load balancer
k8s-cp-01.lab.local Control Plane 1
k8s-cp-02.lab.local Control Plane 2
k8s-cp-03.lab.local Control Plane 3
registry.lab.local internal registry
*.apps.lab.local Ingress/Gateway wildcard, if used
07 / 39

8. Operating System Baseline

8.1. Shared Linux Preparation

These steps apply to classic Linux hosts with kubeadm, k3s, RKE2, or k0s.

bash
sudo hostnamectl set-hostname <NODE_NAME>

sudo timedatectl set-ntp true
sudo systemctl enable --now systemd-timesyncd || true

ip addr
ip route
getent hosts k8s-api.lab.local || true

Validate unique host identity:

bash
hostname -f || hostname
ip link
sudo cat /sys/class/dmi/id/product_uuid || true

8.2. Swap

For classic kubeadm installations, disable swap unless you intentionally configure kubelet swap behavior.

bash
sudo swapoff -a
sudo sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab
free -h

8.3. Kernel Modules and sysctl

bash
cat <<'EOF' | sudo tee /etc/modules-load.d/kubernetes.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

Validation:

bash
lsmod | grep -E 'overlay|br_netfilter'
sysctl net.bridge.bridge-nf-call-iptables
sysctl net.ipv4.ip_forward

8.4. Base Packages

Ubuntu/Debian:

bash
sudo apt-get update
sudo apt-get install -y \
  ca-certificates \
  curl \
  gnupg \
  lsb-release \
  apt-transport-https \
  jq \
  yq \
  git \
  vim \
  bash-completion \
  conntrack \
  socat \
  ebtables \
  ethtool \
  iproute2 \
  iptables \
  nftables \
  nfs-common \
  open-iscsi

RHEL/Rocky/Alma/Fedora:

bash
sudo dnf install -y \
  ca-certificates \
  curl \
  gnupg2 \
  jq \
  git \
  vim \
  bash-completion \
  conntrack-tools \
  socat \
  ebtables \
  ethtool \
  iproute \
  iptables \
  nftables \
  nfs-utils \
  iscsi-initiator-utils

SLES/openSUSE:

bash
sudo zypper refresh
sudo zypper install -y \
  ca-certificates \
  curl \
  gpg2 \
  jq \
  git \
  vim \
  bash-completion \
  conntrack-tools \
  socat \
  ebtables \
  ethtool \
  iproute2 \
  iptables \
  nftables \
  nfs-client \
  open-iscsi

8.5. SELinux and AppArmor

On RHEL-compatible systems, handle SELinux deliberately. For simple kubeadm labs, permissive is often used. For production environments, do not disable SELinux blindly; align CNI, CSI, and runtime with it.

bash
getenforce || true
sudo setenforce 0 || true
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config || true

On Ubuntu/Debian, check AppArmor:

bash
sudo aa-status || true

8.6. Firewall

For labs, temporarily disabling host firewalls can be useful. For production-like environments, document the allowed ports instead.

Ubuntu with UFW:

bash
sudo ufw status verbose
# Lab only, and only if you understand the effect:
# sudo ufw disable

RHEL/Rocky with firewalld:

bash
sudo firewall-cmd --state || true
sudo firewall-cmd --list-all || true

Example openings for the control plane:

bash
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10257/tcp
sudo firewall-cmd --permanent --add-port=10259/tcp
sudo firewall-cmd --reload
08 / 39

9. Container Runtime

Kubernetes talks to container runtimes through CRI. Typical bare-metal options are containerd and CRI-O.

9.1. Runtime Decision

Runtime Suitable for Notes
containerd standard labs, production, k3s/RKE2 internally widely used, lightweight
CRI-O OpenShift-like or Red-Hat-like environments strongly Kubernetes-focused
Docker Engine + cri-dockerd legacy scenarios additional shim required, not preferred

9.2. containerd on Ubuntu/Debian

Variant through the distribution:

bash
sudo apt-get update
sudo apt-get install -y containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl enable --now containerd
sudo systemctl restart containerd
sudo systemctl status containerd --no-pager

Validation:

bash
sudo ctr version
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock info || true

9.3. containerd on RHEL/Rocky/Alma/Fedora

bash
sudo dnf install -y containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl enable --now containerd
sudo systemctl restart containerd

If the distribution does not provide a suitable containerd, use an approved package source based on your internal standard. Document source, version, and upgrade path.

9.4. CRI-O Baseline

CRI-O versions should match the Kubernetes minor version. Define the version centrally.

bash
export KUBERNETES_MINOR="v1.36"
export CRIO_VERSION="v1.36"

The concrete repository configuration depends on distribution and project standard. After installation, validate:

bash
sudo systemctl enable --now crio
sudo systemctl status crio --no-pager
sudo crictl --runtime-endpoint unix:///var/run/crio/crio.sock info

9.5. Configure crictl

For containerd:

bash
cat <<'EOF' | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

For CRI-O:

bash
cat <<'EOF' | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/crio/crio.sock
image-endpoint: unix:///var/run/crio/crio.sock
timeout: 10
debug: false
EOF

Validation:

bash
sudo crictl info
sudo crictl images

9.6. Registry Mirrors and Private Registries

Example for containerd with an internal mirror:

toml
# /etc/containerd/config.toml, excerpt
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://registry.lab.local:5000"]

[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.lab.local:5000".tls]
  ca_file = "/etc/containerd/certs.d/registry.lab.local/ca.crt"

Then:

bash
sudo systemctl restart containerd
sudo crictl pull registry.k8s.io/pause:3.10 || true
09 / 39

10. Install kubeadm

10.1. Pin the Version

Define the Kubernetes minor version centrally:

bash
export KUBERNETES_MINOR="v1.36"

Change this value deliberately if your workshop or project uses a different version. Do not mix minor versions without an upgrade plan.

10.2. Ubuntu/Debian with pkgs.k8s.io

bash
export KUBERNETES_MINOR="v1.36"

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings

curl -fsSL "https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/deb/Release.key" \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

10.3. RHEL/Rocky/Alma/Fedora with pkgs.k8s.io

bash
export KUBERNETES_MINOR="v1.36"

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

With DNF5, --setopt=disable_excludes=kubernetes may be required.

10.4. Without a Package Manager

For distributions without supported package sources, use binaries. This is more advanced and must be versioned cleanly.

bash
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
ARCH="amd64"
DOWNLOAD_DIR="/usr/local/bin"

sudo mkdir -p "${DOWNLOAD_DIR}"
cd "${DOWNLOAD_DIR}"
sudo curl -L --remote-name-all "https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl}"
sudo chmod +x kubeadm kubelet kubectl

You also need a suitable kubelet systemd unit. For labs, the package-based path is much simpler.

10.5. Validate the Installation

bash
kubeadm version
kubectl version --client
kubelet --version
sudo systemctl status kubelet --no-pager

It is normal that kubelet is not fully healthy before kubeadm init. It is waiting for configuration from kubeadm.

10 / 39

11. kubeadm: Single-Node Cluster

11.1. Create the Configuration File

For a reproducible setup, use a kubeadm configuration file.

bash
mkdir -p ~/kubeadm
cat <<'EOF' > ~/kubeadm/kubeadm-single-node.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.20.11
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: 10.10.20.11
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: stable
clusterName: bm-single
controlPlaneEndpoint: 10.10.20.11:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
  dnsDomain: cluster.local
apiServer:
  certSANs:
    - 10.10.20.11
    - bm-single.lab.local
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

Adapt advertiseAddress, node-ip, clusterName, podSubnet, serviceSubnet, and certSANs to your environment.

11.2. Initialize the Control Plane

bash
sudo kubeadm init --config ~/kubeadm/kubeadm-single-node.yaml

Set up kubeconfig:

bash
mkdir -p "$HOME/.kube"
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

kubectl cluster-info
kubectl get nodes -o wide

11.3. Allow Workloads on the Control Plane

Single-node labs need workloads on the control-plane node. Remove the taint deliberately:

bash
kubectl taint nodes --all node-role.kubernetes.io/control-plane- || true
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

11.4. Install the CNI

Without a CNI, nodes typically remain NotReady, and CoreDNS does not start fully.

Example with Flannel and matching pod CIDR:

bash
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Example with the Cilium CLI:

bash
cilium install --version <CILIUM_VERSION>
cilium status --wait

Example with Calico:

bash
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/<CALICO_VERSION>/manifests/calico.yaml

Choose exactly one primary CNI plugin. Multiple active CNIs without a clear chaining model create difficult-to-debug failures.

11.5. Validate

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n kube-system get pods
kubectl run smoke --image=registry.k8s.io/e2e-test-images/agnhost:2.53 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/smoke --timeout=120s
kubectl delete pod smoke
11 / 39

12. kubeadm: Multi-Node Cluster

12.1. Initialize the Control Plane

Use a stable control-plane endpoint even if you start with one control-plane node. This keeps a later HA expansion possible.

yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.20.11
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: 10.10.20.11
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: stable
clusterName: bm-multi
controlPlaneEndpoint: k8s-api.lab.local:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - k8s-api.lab.local
    - 10.10.20.10
    - 10.10.20.11
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Initialization:

bash
sudo kubeadm init --config kubeadm-multi.yaml

12.2. Generate the Join Command

If you did not save the join command:

bash
kubeadm token create --print-join-command

Example:

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

12.3. Prepare Workers and Join Them

On every worker:

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

Then on your admin client:

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide

12.4. Set Labels

bash
kubectl label node k8s-w-01 node-role.kubernetes.io/worker=worker
kubectl label node k8s-w-02 node-role.kubernetes.io/worker=worker
kubectl label node k8s-w-03 node-role.kubernetes.io/worker=worker

kubectl label node k8s-w-01 topology.kubernetes.io/zone=rack-a
kubectl label node k8s-w-02 topology.kubernetes.io/zone=rack-b
kubectl label node k8s-w-03 topology.kubernetes.io/zone=rack-c

12.5. Smoke Test with Multiple Nodes

bash
kubectl create namespace smoke
kubectl create deployment web -n smoke --image=nginx:stable --replicas=3
kubectl expose deployment web -n smoke --port=80 --target-port=80
kubectl get pod -n smoke -o wide
kubectl run curl -n smoke --rm -it --image=curlimages/curl --restart=Never -- curl -sS http://web.smoke.svc.cluster.local
kubectl delete namespace smoke
12 / 39

13. kubeadm: High Availability

13.1. Requirements

For control-plane HA, you need:

  • at least three control-plane nodes,
  • a stable API endpoint in front of the control-plane nodes,
  • a load balancer for TCP 6443,
  • consistent certificate SANs,
  • working name resolution,
  • identical Kubernetes minor version on all nodes,
  • working time synchronization,
  • an etcd backup strategy.

13.2. Load-Balancer Options

Option Advantage Disadvantage Typical use
Hardware load balancer robust, centrally managed depends on network team data center
HAProxy + Keepalived transparent, inexpensive two additional hosts or runs on CP nodes lab, mid-sized environments
kube-vip Kubernetes-native, no external appliance must be planned very carefully small HA clusters
BGP LB scalable, elegant router/BGP expertise required network-centric platforms
DNS round-robin simple no real health checking tests only

13.3. HAProxy Example

/etc/haproxy/haproxy.cfg:

cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon
    maxconn 4096

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 10s
    timeout client  1m
    timeout server  1m

frontend kubernetes_api
    bind *:6443
    default_backend kubernetes_control_plane

backend kubernetes_control_plane
    balance roundrobin
    option tcp-check
    server cp1 10.10.20.11:6443 check
    server cp2 10.10.20.12:6443 check
    server cp3 10.10.20.13:6443 check

Start:

bash
sudo systemctl enable --now haproxy
sudo systemctl status haproxy --no-pager
nc -vz k8s-api.lab.local 6443

13.4. Keepalived Example

/etc/keepalived/keepalived.conf on the primary LB:

cfg
vrrp_instance K8S_API {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 110
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass CHANGE_ME
    }
    virtual_ipaddress {
        10.10.20.10/24
    }
}

On the second LB, set state BACKUP and a lower priority.

bash
sudo systemctl enable --now keepalived
ip addr show eth0

13.5. kube-vip as the Control-Plane VIP

kube-vip can provide a virtual IP for the Kubernetes API. This is particularly useful in smaller bare-metal clusters without an external load balancer.

Concept:

flowchart LR
    Client[kubectl] --> VIP[VIP 10.10.20.10]
    VIP --> CP1[kube-vip on cp1]
    VIP --> CP2[kube-vip on cp2]
    VIP --> CP3[kube-vip on cp3]
    CP1 --> API1[API Server cp1]
    CP2 --> API2[API Server cp2]
    CP3 --> API3[API Server cp3]

Important:

  • Generate the kube-vip manifest before kubeadm init if kube-vip should run as a static pod.
  • Decide whether to use ARP, BGP, or routing mode.
  • Make sure the VIP is not already allocated elsewhere.

13.6. HA kubeadm Configuration

kubeadm-ha.yaml on the first control-plane node:

yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.20.11
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: 10.10.20.11
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: stable
clusterName: bm-ha
controlPlaneEndpoint: k8s-api.lab.local:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - k8s-api.lab.local
    - 10.10.20.10
    - 10.10.20.11
    - 10.10.20.12
    - 10.10.20.13
controllerManager: {}
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Initialization:

bash
sudo kubeadm init --config kubeadm-ha.yaml --upload-certs

Store the emitted join commands securely. The certificate key from --upload-certs is time-limited.

13.7. Join Additional Control-Plane Nodes

On k8s-cp-02 and k8s-cp-03:

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane \
  --certificate-key <CERTIFICATE_KEY>

After each join, validate:

bash
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
kubectl get --raw='/readyz?verbose'

13.8. Join Workers

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

13.9. Redistribute CoreDNS After HA Expansion

If you add control-plane nodes one after another, CoreDNS pods can initially be concentrated on the first node.

bash
kubectl -n kube-system rollout restart deployment coredns
kubectl -n kube-system rollout status deployment coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
13 / 39

14. External etcd

External etcd is useful when you want to operate etcd resources and the control plane separately.

14.1. Architecture

flowchart TD
    LB[API Load Balancer] --> CP1[API Server 1]
    LB --> CP2[API Server 2]
    LB --> CP3[API Server 3]
    CP1 --> ETCDLB[etcd Endpoints]
    CP2 --> ETCDLB
    CP3 --> ETCDLB
    ETCDLB --> E1[etcd 1]
    ETCDLB --> E2[etcd 2]
    ETCDLB --> E3[etcd 3]

14.2. Requirements

  • Three or five etcd members.
  • Low latency between etcd members.
  • Fast SSDs with stable I/O latency.
  • Secured peer and client certificates.
  • Encrypted backups.
  • Regular restore tests.

14.3. kubeadm Configuration Excerpt

yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: k8s-api.lab.local:6443
etcd:
  external:
    endpoints:
      - https://10.10.40.11:2379
      - https://10.10.40.12:2379
      - https://10.10.40.13:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
14 / 39

15. CNI Selection and Installation

The CNI determines pod networking, NetworkPolicy, routing, overlay, eBPF, encryption, and sometimes load balancing.

15.1. Decision Matrix

CNI Strengths Typical use Notes
Flannel simple, easy to explain workshops, small labs no native NetworkPolicy enforcement
Calico NetworkPolicy, BGP, robust labs, production very common
Cilium eBPF, NetworkPolicy, observability, kube-proxy replacement modern platforms check kernel and mode requirements
Weave Net historically simple legacy check project status
Canal Flannel + Calico policy certain legacy setups less common today

15.2. Flannel

Matching podSubnet is usually 10.244.0.0/16.

bash
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl -n kube-flannel get pods -o wide

15.3. Calico

Calico can use overlay or BGP. For bare metal, BGP is interesting but requires coordination with router/network configuration.

bash
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/<CALICO_VERSION>/manifests/calico.yaml
kubectl -n calico-system get pods -o wide || kubectl -n kube-system get pods -o wide | grep calico

15.4. Cilium

Generic installation path:

bash
cilium install --version <CILIUM_VERSION>
cilium status --wait
cilium connectivity test

With kube-proxy replacement, initialize the cluster accordingly, for example with --skip-phases=addon/kube-proxy or an appropriate kubeadm configuration. Check kernel, cgroup v2, routing mode, LoadBalancer mode, and Hubble requirements first.

15.5. CNI Validation

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n kube-system get daemonsets
kubectl run net-a --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl run net-b --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/net-a --timeout=120s
kubectl wait --for=condition=Ready pod/net-b --timeout=120s
kubectl get pods -o wide
kubectl exec net-a -- ping -c 3 <POD_IP_OF_NET_B>
kubectl delete pod net-a net-b
15 / 39

16. Services, LoadBalancer, and Ingress on Bare Metal

In cloud environments, the cloud controller often provides external load balancers. On bare metal, you must provide this function yourself.

16.1. Options

Function Option Note
Service LoadBalancer MetalLB L2 simple, Layer-2-oriented
Service LoadBalancer MetalLB BGP better for scalable networks
API VIP and Services kube-vip good for small to medium bare-metal clusters
Edge Ingress NodePort + external reverse proxy simple, but manual
Production Ingress hardware/software LB + Ingress Controller robust, established operational boundaries

16.2. MetalLB Layer 2

Install using manifests or Helm based on project standards. Then define an IP address pool and advertisement.

yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.10.20.200-10.10.20.240
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - lab-pool

Validation:

bash
kubectl get pods -n metallb-system
kubectl apply -f metallb-pool.yaml
kubectl create deployment lb-test --image=nginx:stable
kubectl expose deployment lb-test --port=80 --type=LoadBalancer
kubectl get svc lb-test -w
curl http://<EXTERNAL_IP>
kubectl delete svc lb-test
kubectl delete deployment lb-test

16.3. MetalLB BGP

BGP is often cleaner for larger bare-metal environments. You need routers, an ASN plan, and coordinated policies.

Example resources:

yaml
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: router-a
  namespace: metallb-system
spec:
  myASN: 64512
  peerASN: 64500
  peerAddress: 10.10.20.1
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: bgp-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.10.50.0/24
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgp-adv
  namespace: metallb-system
spec:
  ipAddressPools:
    - bgp-pool

16.4. Ingress Controller

Example with NGINX Ingress Controller through Helm:

bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=LoadBalancer

Validation:

bash
kubectl -n ingress-nginx get svc,pods
kubectl get ingressclass

16.5. Gateway API

For modern routing models, you can use Gateway API. You need a controller that supports Gateway API, such as Envoy Gateway, Cilium Gateway API, or another implementation.

Typical workflow:

bash
kubectl get gatewayclass
kubectl get gateways -A
kubectl get httproutes -A
16 / 39

17. Storage on Bare Metal

Storage is an explicit architecture decision on bare metal. Kubernetes provides the API, but not automatically a highly available storage backend.

17.1. Options

Storage Suitable for Advantages Risks
emptyDir temporary data simple data loss on pod restart/node move
hostPath labs, local tests fast insecure, node-bound
Local Persistent Volumes local performance transparent workload bound to node
NFS CSI simple RWX shares easy integration single point without HA NFS
Longhorn small to medium bare-metal clusters simple replicated block storage consider network and disk I/O
OpenEBS local or replicated storage flexible mode-dependent
Rook/Ceph robust platforms scalable, RBD/CephFS/S3 more complex operations
Vendor CSI SAN/NAS/array enterprise features vendor dependency

17.2. Minimal StorageClass Test

bash
kubectl get storageclass
kubectl get csidrivers
kubectl get pv,pvc -A

PVC test:

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-smoke
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: <STORAGE_CLASS>
---
apiVersion: v1
kind: Pod
metadata:
  name: storage-smoke
  namespace: default
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "echo ok > /data/test.txt && sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: storage-smoke

Validation:

bash
kubectl apply -f storage-smoke.yaml
kubectl wait --for=condition=Ready pod/storage-smoke --timeout=180s
kubectl exec storage-smoke -- cat /data/test.txt
kubectl delete -f storage-smoke.yaml

17.3. Storage Design Questions

  • Do you need ReadWriteOnce, ReadWriteMany, or ReadOnlyMany?
  • May a workload start automatically on another node after node failure?
  • How is volume replication implemented?
  • How are snapshots created?
  • How do you test restore?
  • Which data is cache only, and which data is business-critical?
  • Who operates the storage backend?
  • What latency and IOPS requirements apply?
17 / 39

18. Add-ons After the Base Installation

Install add-ons only after the API, CNI, nodes, and DNS are healthy.

18.1. Order

flowchart TD
    API[API and Nodes Ready] --> CNI[CNI]
    CNI --> DNS[CoreDNS healthy]
    DNS --> Metrics[Metrics Server]
    Metrics --> LB[MetalLB or kube-vip Services]
    LB --> Ingress[Ingress or Gateway]
    Ingress --> Certs[cert-manager]
    Certs --> Storage[CSI / Storage]
    Storage --> Obs[Observability]
    Obs --> GitOps[GitOps Bootstrap]
    GitOps --> Policies[Policies and Security]

18.2. Metrics Server

bash
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system rollout status deployment metrics-server
kubectl top nodes
kubectl top pods -A

In bare-metal labs with self-signed kubelet certificates, additional configuration may be required. For production, solve certificates correctly instead of adopting insecure workarounds permanently.

18.3. cert-manager

bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

kubectl -n cert-manager get pods

18.4. Observability Baseline

For workshops, this is often enough:

  • Metrics Server,
  • kube-state-metrics,
  • Prometheus,
  • Grafana,
  • Loki or another logging system,
  • optionally Tempo or OpenTelemetry Collector.

Validation commands:

bash
kubectl get pods -A | grep -E 'prometheus|grafana|loki|otel|tempo|metrics'
kubectl get servicemonitors -A 2>/dev/null || true
kubectl get podmonitors -A 2>/dev/null || true

18.5. GitOps Bootstrap

Example order:

  1. Provision the cluster base with kubeadm or a distribution.
  2. Install CNI.
  3. Provide LoadBalancer functionality.
  4. Install the GitOps controller.
  5. Connect add-on repositories.
  6. Synchronize policies and applications from Git.

Argo CD example:

bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deployment/argocd-server

Flux example:

bash
flux check --pre
flux bootstrap git \
  --url=ssh://git@example.local/platform/kubernetes.git \
  --branch=main \
  --path=clusters/bm-lab-01
18 / 39

19. Security Baseline

19.1. Host Hardening

  • Install only necessary packages.
  • Restrict SSH to key-based authentication.
  • Disable root login through SSH.
  • Separate the BMC network.
  • Enforce NTP.
  • Deliberately govern automatic security updates.
  • Enable audit logs.
  • Do not make container and kubelet directories broadly writable.
  • Configure and document the host firewall.

SSH example:

bash
sudo sed -i 's/^#\?PermitRootLogin .*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?PasswordAuthentication .*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl reload ssh || sudo systemctl reload sshd

19.2. Protect the Kubernetes API

  • Do not share admin.conf.
  • Solve user access through individual certificates, OIDC, or external identity.
  • Create RBAC with least privilege.
  • Do not use system:masters for daily access.
  • Plan audit logging.
  • Use admission policies.

Validation:

bash
kubectl auth can-i get pods -A
kubectl auth can-i '*' '*' --as system:serviceaccount:default:default
kubectl get clusterrolebinding -o wide

19.3. Pod Security Admission

Set namespace labels:

bash
kubectl label namespace default pod-security.kubernetes.io/enforce=baseline --overwrite
kubectl label namespace default pod-security.kubernetes.io/audit=restricted --overwrite
kubectl label namespace default pod-security.kubernetes.io/warn=restricted --overwrite

For production-like workloads, prefer restricted, but align images, SecurityContext, and volumes with it.

19.4. Encryption at Rest

Without additional configuration, Secrets are stored as Kubernetes objects. Enable encryption at rest for production-like clusters.

Example EncryptionConfiguration:

yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64_32_BYTE_KEY>
      - identity: {}

You must adapt the API server configuration accordingly and manage the key securely. Test rotation and restore.

19.5. NetworkPolicies

Start with a default-deny model per namespace and allow only required communication.

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

DNS exception:

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
19 / 39

20. Air-Gap and Restricted Environments

Air-gap does not simply mean “no internet.” It means every package, image, chart, key, and signature must be provided in a controlled way.

20.1. Artifact Groups

Group Examples
OS packages containerd, kubelet, kubeadm, kubectl, cri-tools
Container images control plane, CoreDNS, kube-proxy, pause, CNI, add-ons
Helm charts CNI, Ingress, cert-manager, observability
Manifests MetalLB, policies, storage, GitOps
Signatures/SBOMs Cosign, checksums, provenance
Certificates internal CA, registry CA, TLS certificates

20.2. Workflow

flowchart LR
    Internet[Online Staging] --> Mirror[Artifact Mirror]
    Mirror --> Scan[Scan, Signature Verification, Approval]
    Scan --> Transfer[Controlled Transfer]
    Transfer --> Registry[Internal Registry]
    Transfer --> Repo[Internal Package Source]
    Registry --> Nodes[Bare-Metal Nodes]
    Repo --> Nodes
    Nodes --> Cluster[Kubernetes Cluster]

20.3. List Images for kubeadm

bash
kubeadm config images list --kubernetes-version <VERSION>

Pre-pull images:

bash
sudo kubeadm config images pull --kubernetes-version <VERSION> --cri-socket unix:///run/containerd/containerd.sock

With an alternative registry:

bash
sudo kubeadm config images pull \
  --kubernetes-version <VERSION> \
  --image-repository registry.lab.local/kubernetes

20.4. Image Rewrite in kubeadm

yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.0
imageRepository: registry.lab.local/kubernetes

20.5. Test the Local Registry

bash
curl -k https://registry.lab.local/v2/_catalog || true
sudo crictl pull registry.lab.local/kubernetes/pause:3.10

20.6. Air-Gap Checklist

  • All OS packages are available in an internal package source.
  • Kubernetes minor version is pinned.
  • All container images are mirrored.
  • All Helm charts are internally available.
  • Registry CA is installed on all nodes.
  • Runtime uses the registry mirror.
  • CNI manifests reference internal images.
  • Add-on manifests reference internal images.
  • Installation has been tested without internet access.
  • Restore procedure works offline.
20 / 39

21. k3s on Bare Metal

k3s is a lightweight Kubernetes distribution. It is well suited for edge, small bare-metal clusters, ARM systems, and labs.

21.1. Single-Node k3s

bash
curl -sfL https://get.k3s.io | sh -

sudo k3s kubectl get nodes
sudo cat /etc/rancher/k3s/k3s.yaml

Kubeconfig for your user:

bash
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
kubectl get nodes

21.2. k3s with Pinned Options

bash
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="\
  --cluster-cidr=10.42.0.0/16 \
  --service-cidr=10.43.0.0/16 \
  --disable traefik \
  --disable servicelb" sh -

Why disable add-ons?

  • You want to install your own Ingress and LoadBalancer variants in the workshop.
  • You want to explain behavior transparently.
  • You want to manage add-ons through GitOps.

21.3. k3s HA with Embedded etcd

First server:

bash
curl -sfL https://get.k3s.io | K3S_TOKEN="<CLUSTER_SECRET>" sh -s - server \
  --cluster-init \
  --tls-san k3s-api.lab.local \
  --disable traefik \
  --disable servicelb

Additional servers:

bash
curl -sfL https://get.k3s.io | K3S_TOKEN="<CLUSTER_SECRET>" sh -s - server \
  --server https://k3s-api.lab.local:6443 \
  --tls-san k3s-api.lab.local \
  --disable traefik \
  --disable servicelb

Agent:

bash
curl -sfL https://get.k3s.io | K3S_TOKEN="<CLUSTER_SECRET>" sh -s - agent \
  --server https://k3s-api.lab.local:6443

21.4. Validate k3s

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
sudo systemctl status k3s --no-pager
sudo journalctl -u k3s -n 100 --no-pager
21 / 39

22. RKE2 on Bare Metal

RKE2 is a Kubernetes distribution from Rancher/SUSE with a focus on security and enterprise environments.

22.1. First Server

bash
curl -sfL https://get.rke2.io | sudo sh -
sudo mkdir -p /etc/rancher/rke2
cat <<'EOF' | sudo tee /etc/rancher/rke2/config.yaml
token: <CLUSTER_SECRET>
tls-san:
  - rke2-api.lab.local
node-ip: 10.10.20.11
cni: cilium
EOF

sudo systemctl enable --now rke2-server

Kubeconfig:

bash
mkdir -p ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
export PATH=$PATH:/var/lib/rancher/rke2/bin
kubectl get nodes

22.2. Additional Servers

bash
curl -sfL https://get.rke2.io | sudo sh -
sudo mkdir -p /etc/rancher/rke2
cat <<'EOF' | sudo tee /etc/rancher/rke2/config.yaml
server: https://rke2-api.lab.local:9345
token: <CLUSTER_SECRET>
tls-san:
  - rke2-api.lab.local
node-ip: 10.10.20.12
cni: cilium
EOF

sudo systemctl enable --now rke2-server

22.3. Agent Nodes

bash
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_TYPE="agent" sh -
sudo mkdir -p /etc/rancher/rke2
cat <<'EOF' | sudo tee /etc/rancher/rke2/config.yaml
server: https://rke2-api.lab.local:9345
token: <CLUSTER_SECRET>
node-ip: 10.10.20.21
EOF

sudo systemctl enable --now rke2-agent

22.4. Validate RKE2

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
sudo journalctl -u rke2-server -n 100 --no-pager
sudo journalctl -u rke2-agent -n 100 --no-pager || true
22 / 39

23. k0s on Bare Metal

k0s provides a simple controller/worker model and bundles many components.

23.1. Single-Node k0s

bash
curl -sSfL https://get.k0s.sh | sudo sh
sudo k0s install controller --single
sudo k0s start
sudo k0s status
sudo k0s kubectl get nodes

Kubeconfig:

bash
mkdir -p ~/.kube
sudo k0s kubeconfig admin > ~/.kube/config
chmod 600 ~/.kube/config
kubectl get nodes

23.2. Multi-Node k0s

Install controller:

bash
sudo k0s install controller
sudo k0s start
sudo k0s token create --role=worker > worker-token
sudo k0s token create --role=controller > controller-token

Install worker:

bash
curl -sSfL https://get.k0s.sh | sudo sh
sudo k0s install worker --token-file /path/to/worker-token
sudo k0s start

23.3. Validate k0s

bash
sudo k0s status
sudo k0s kubectl get nodes -o wide
sudo k0s kubectl get pods -A -o wide
23 / 39

24. Talos Linux on Bare Metal

Talos is a specialized Kubernetes operating system. You manage nodes through the Talos API with talosctl, not through SSH.

24.1. Workflow

flowchart TD
    ISO[Boot Talos ISO] --> Maintenance[Node in Maintenance Mode]
    Maintenance --> Generate[talosctl gen config]
    Generate --> ApplyCP[Apply Control-Plane Configuration]
    Generate --> ApplyWorker[Apply Worker Configuration]
    ApplyCP --> Bootstrap[talosctl bootstrap once]
    Bootstrap --> Kubeconfig[talosctl kubeconfig]
    Kubeconfig --> Validate[kubectl get nodes]

24.2. Generate Configuration

bash
export CLUSTER_NAME="talos-bm-lab"
export CONTROL_PLANE_ENDPOINT="https://10.10.20.10:6443"

talosctl gen config "${CLUSTER_NAME}" "${CONTROL_PLANE_ENDPOINT}" --output-dir ./talos-out

24.3. Apply Configuration

Control plane:

bash
talosctl apply-config \
  --insecure \
  --nodes 10.10.20.11 \
  --file ./talos-out/controlplane.yaml

Worker:

bash
talosctl apply-config \
  --insecure \
  --nodes 10.10.20.21 \
  --file ./talos-out/worker.yaml

24.4. Bootstrap

Bootstrap is executed exactly once on one control-plane node:

bash
talosctl bootstrap \
  --nodes 10.10.20.11 \
  --endpoints 10.10.20.11 \
  --talosconfig ./talos-out/talosconfig

Retrieve kubeconfig:

bash
talosctl kubeconfig \
  --nodes 10.10.20.11 \
  --endpoints 10.10.20.11 \
  --talosconfig ./talos-out/talosconfig \
  --force

kubectl get nodes -o wide

24.5. Talos Debugging

bash
talosctl health --nodes 10.10.20.11 --endpoints 10.10.20.11
talosctl service kubelet --nodes 10.10.20.11
talosctl logs kubelet --nodes 10.10.20.11
talosctl dmesg --nodes 10.10.20.11
talosctl disks --nodes 10.10.20.11
24 / 39

25. Automation with Ansible

For repeatable bare-metal provisioning, move manual commands into roles.

25.1. Inventory

ini
[control_plane]
k8s-cp-01 ansible_host=10.10.20.11 node_ip=10.10.20.11
k8s-cp-02 ansible_host=10.10.20.12 node_ip=10.10.20.12
k8s-cp-03 ansible_host=10.10.20.13 node_ip=10.10.20.13

[workers]
k8s-w-01 ansible_host=10.10.20.21 node_ip=10.10.20.21
k8s-w-02 ansible_host=10.10.20.22 node_ip=10.10.20.22
k8s-w-03 ansible_host=10.10.20.23 node_ip=10.10.20.23

[kubernetes:children]
control_plane
workers

25.2. Role Structure

text
ansible/
  inventory.ini
  group_vars/
    all.yaml
  playbooks/
    00-preflight.yaml
    10-os-baseline.yaml
    20-container-runtime.yaml
    30-kubernetes-packages.yaml
    40-kubeadm-init.yaml
    50-kubeadm-join.yaml
    60-addons.yaml
    90-validate.yaml
  roles/
    os_baseline/
    containerd/
    kubeadm/
    cni/
    metallb/
    ingress/
    observability/

25.3. Example: OS Baseline Tasks

yaml
- name: Load Kubernetes kernel modules
  ansible.builtin.copy:
    dest: /etc/modules-load.d/kubernetes.conf
    content: |
      overlay
      br_netfilter

- name: Load modules immediately
  ansible.builtin.command: modprobe {{ item }}
  loop:
    - overlay
    - br_netfilter
  changed_when: false

- name: Set sysctl
  ansible.builtin.copy:
    dest: /etc/sysctl.d/99-kubernetes-cri.conf
    content: |
      net.bridge.bridge-nf-call-iptables = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward = 1

- name: Apply sysctl
  ansible.builtin.command: sysctl --system
  changed_when: false

25.4. Example: Handle the Join File Securely

  • Do not store tokens in plaintext in Git.
  • Generate join commands only temporarily.
  • Use the certificate key from --upload-certs only briefly.
  • Manage secrets in Ansible Vault, SOPS, or a secret manager.
25 / 39

26. PXE, Autoinstall, and Metal3

26.1. Provisioning Chain

sequenceDiagram
    participant Admin
    participant BMC as BMC/IPMI/Redfish
    participant DHCP
    participant PXE as PXE/iPXE
    participant HTTP as HTTP Artifact Server
    participant Node
    participant Ansible
    participant K8s as Kubernetes API

    Admin->>BMC: Power cycle and set boot device
    Node->>DHCP: Request lease and boot options
    DHCP->>Node: iPXE/PXE information
    Node->>PXE: Load bootloader
    Node->>HTTP: Load kernel, initrd, autoinstall
    Node->>Node: Install OS
    Ansible->>Node: Install baseline, runtime, kubeadm
    Ansible->>K8s: Initialize cluster and join nodes

26.2. Ubuntu Autoinstall

An Ubuntu autoinstall file defines users, network, storage, and packages. Example excerpt:

yaml
# user-data, heavily shortened
autoinstall:
  version: 1
  identity:
    hostname: k8s-cp-01
    username: ubuntu
    password: "$6$REPLACE_WITH_HASH"
  ssh:
    install-server: true
    authorized-keys:
      - ssh-ed25519 AAAA... workshop
  packages:
    - curl
    - git
    - jq
    - vim
  late-commands:
    - curtin in-target --target=/target -- systemctl enable ssh

26.3. Kickstart for RHEL-Compatible Systems

cfg
text
reboot
network --bootproto=dhcp --device=link --activate
rootpw --iscrypted <HASH>
user --name=admin --groups=wheel --iscrypted --password=<HASH>
%packages
@^minimal-environment
curl
git
jq
vim
%end
%post
systemctl enable sshd
%end

26.4. Metal3 and Cluster API

Metal3 models physical servers as Kubernetes resources. This is useful if you want to manage bare-metal clusters declaratively and repeatably through Cluster API.

Typical building blocks:

  • management cluster,
  • BareMetalHost resources,
  • BMC credentials,
  • Ironic,
  • image server,
  • Cluster API Provider Metal3,
  • GitOps for cluster definitions.

This is powerful, but usually too complex for an introductory workshop. For platform teams, it is a good advanced module.

26 / 39

27. End-to-End Example: kubeadm HA on Ubuntu Bare Metal

This chapter summarizes the recommended learning path as a complete workflow.

27.1. Assumptions

Value Example
Cluster bm-ha
Kubernetes minor v1.36
Control plane 10.10.20.11-13
Workers 10.10.20.21-23
API VIP 10.10.20.10
API DNS k8s-api.lab.local
Pod CIDR 10.244.0.0/16
Service CIDR 10.96.0.0/12
Runtime containerd
CNI Flannel or Cilium, depending on lab objective
LoadBalancer MetalLB L2
Ingress ingress-nginx

27.2. On All Nodes: Prepare the OS

bash
sudo swapoff -a
sudo sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab

cat <<'EOF' | sudo tee /etc/modules-load.d/kubernetes.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

27.3. On All Nodes: Install containerd

bash
sudo apt-get update
sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl enable --now containerd
sudo systemctl restart containerd

27.4. On All Nodes: Install kubeadm

bash
export KUBERNETES_MINOR="v1.36"

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL "https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/deb/Release.key" \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

27.5. Prepare the Load Balancer

Make sure k8s-api.lab.local points to 10.10.20.10 and forwards TCP 6443 to all control-plane nodes.

bash
getent hosts k8s-api.lab.local
nc -vz k8s-api.lab.local 6443 || true

Before kubeadm init, port 6443 may still be closed. After the first API server starts, the check must succeed.

27.6. Initialize the First Control-Plane Node

bash
cat <<'EOF' > kubeadm-ha.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.20.11
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: 10.10.20.11
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: stable
clusterName: bm-ha
controlPlaneEndpoint: k8s-api.lab.local:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - k8s-api.lab.local
    - 10.10.20.10
    - 10.10.20.11
    - 10.10.20.12
    - 10.10.20.13
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

sudo kubeadm init --config kubeadm-ha.yaml --upload-certs

Kubeconfig:

bash
mkdir -p "$HOME/.kube"
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"
kubectl get nodes -o wide

27.7. Install the CNI

Flannel example:

bash
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl -n kube-flannel get pods -w

Cilium example:

bash
cilium install --version <CILIUM_VERSION>
cilium status --wait

27.8. Join Additional Control-Plane Nodes

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane \
  --certificate-key <CERTIFICATE_KEY>

27.9. Join Workers

bash
sudo kubeadm join k8s-api.lab.local:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

27.10. Validate the Cluster

bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n kube-system rollout status deployment coredns
kubectl get --raw='/readyz?verbose'

27.11. Install and Test MetalLB

bash
kubectl get ns metallb-system || true
# Installation according to project standard: Helm, Kustomize, or manifests.

Pool:

yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.10.20.200-10.10.20.240
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - lab-pool

Test:

bash
kubectl create deployment nginx --image=nginx:stable
kubectl expose deployment nginx --port=80 --type=LoadBalancer
kubectl get svc nginx -w
curl http://<EXTERNAL_IP>
kubectl delete svc nginx
kubectl delete deployment nginx
27 / 39

28. Validation Catalog

28.1. Validation Phases

flowchart TD
    V1[Validate Host] --> V2[Validate Runtime]
    V2 --> V3[Validate kubeadm]
    V3 --> V4[Validate API]
    V4 --> V5[Validate CNI]
    V5 --> V6[Validate DNS]
    V6 --> V7[Validate Service]
    V7 --> V8[Validate LoadBalancer]
    V8 --> V9[Validate Storage]
    V9 --> V10[Validate Security]
    V10 --> V11[Validate Upgrade and Teardown Path]

28.2. Host

bash
hostnamectl
uname -a
ip addr
ip route
timedatectl
free -h
swapon --show
lsmod | grep -E 'overlay|br_netfilter'
sysctl net.ipv4.ip_forward

28.3. Runtime

bash
sudo systemctl status containerd --no-pager || true
sudo systemctl status crio --no-pager || true
sudo crictl info
sudo crictl ps -a
sudo crictl images

28.4. Kubernetes Components

bash
kubeadm version
kubelet --version
kubectl version --client
sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 100 --no-pager

28.5. API and Nodes

bash
kubectl cluster-info
kubectl get nodes -o wide
kubectl describe node <NODE>
kubectl get componentstatuses 2>/dev/null || true
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'

28.6. Control Plane Pods

bash
kubectl -n kube-system get pods -o wide
kubectl -n kube-system get pods -l component=kube-apiserver -o wide
kubectl -n kube-system get pods -l component=kube-controller-manager -o wide
kubectl -n kube-system get pods -l component=kube-scheduler -o wide
kubectl -n kube-system get pods -l component=etcd -o wide

28.7. DNS

bash
kubectl run dns-test --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/dns-test --timeout=120s
kubectl exec dns-test -- nslookup kubernetes.default.svc.cluster.local
kubectl exec dns-test -- nslookup kube-dns.kube-system.svc.cluster.local
kubectl delete pod dns-test

28.8. Service Routing

bash
kubectl create namespace svc-test
kubectl create deployment echo -n svc-test --image=registry.k8s.io/e2e-test-images/agnhost:2.53 -- /agnhost netexec --http-port=8080
kubectl expose deployment echo -n svc-test --port=80 --target-port=8080
kubectl run curl -n svc-test --rm -it --image=curlimages/curl --restart=Never -- curl -sS http://echo
kubectl delete namespace svc-test

28.9. LoadBalancer

bash
kubectl create deployment lb-smoke --image=nginx:stable
kubectl expose deployment lb-smoke --port=80 --type=LoadBalancer
kubectl get svc lb-smoke -w
curl http://<EXTERNAL_IP>
kubectl delete svc lb-smoke
kubectl delete deployment lb-smoke

28.10. Storage

bash
kubectl get storageclass
kubectl get pv,pvc -A
kubectl get volumeattachments -A 2>/dev/null || true

28.11. Security

bash
kubectl auth can-i get pods -A
kubectl get serviceaccounts -A
kubectl get roles,rolebindings -A
kubectl get clusterroles,clusterrolebindings
kubectl get ns --show-labels | grep pod-security || true
28 / 39

29. Troubleshooting

29.1. kubelet Does Not Start

Symptoms:

  • Node does not appear in the cluster.
  • systemctl status kubelet shows errors.
  • journalctl -u kubelet reports runtime, cgroup, or swap problems.

Validation:

bash
sudo journalctl -u kubelet -n 200 --no-pager
sudo crictl info
swapon --show
cat /var/lib/kubelet/config.yaml 2>/dev/null || true

Typical causes:

  • Swap is active.
  • Runtime is not running.
  • Runtime and kubelet cgroup drivers do not match.
  • Wrong node IP.
  • Firewall blocks kubelet or API.
  • Certificate or bootstrap token is invalid.

29.2. Node Remains NotReady

bash
kubectl describe node <NODE>
kubectl get pods -A -o wide
kubectl -n kube-system get daemonsets

Typical causes:

  • CNI is not installed.
  • CNI pods crash.
  • Pod CIDR is missing or mismatched.
  • Runtime cannot pull images.
  • DNS or registry is not reachable.
  • Kernel modules or sysctl are missing.

29.3. CoreDNS Does Not Start

bash
kubectl -n kube-system describe deployment coredns
kubectl -n kube-system logs deployment/coredns
kubectl get nodes -o wide

Typical causes:

  • CNI missing.
  • Nodes are NotReady.
  • Upstream DNS on the host is wrong.
  • NetworkPolicy blocks DNS.

29.4. Pods Hang in Pending

bash
kubectl describe pod <POD> -n <NAMESPACE>
kubectl get events -n <NAMESPACE> --sort-by=.lastTimestamp
kubectl describe nodes | grep -A5 -E 'Allocated resources|Taints'

Typical causes:

  • no free resources,
  • taints without matching tolerations,
  • PVC not bound,
  • NodeSelector/Affinity does not match,
  • image architecture does not match the node architecture.

29.5. ImagePullBackOff

bash
kubectl describe pod <POD> -n <NAMESPACE>
sudo crictl pull <IMAGE>

Typical causes:

  • registry not reachable,
  • wrong image name or tag,
  • private registry without pull secret,
  • TLS CA missing,
  • air-gap mirror incomplete.

29.6. Service Has No Endpoints

bash
kubectl get svc,endpoints,endpointslices -n <NAMESPACE>
kubectl get pods -n <NAMESPACE> --show-labels
kubectl describe svc <SERVICE> -n <NAMESPACE>

Typical causes:

  • Service selector does not match pod labels,
  • pods are not Ready,
  • wrong namespace,
  • Readiness probe fails.

29.7. MetalLB Does Not Assign an IP

bash
kubectl -n metallb-system get pods
kubectl -n metallb-system get ipaddresspools,l2advertisements,bgppeers,bgpadvertisements
kubectl describe svc <SERVICE> -n <NAMESPACE>

Typical causes:

  • IP pool missing,
  • IP pool overlaps or is not in the right network,
  • L2/BGP mode does not fit the network environment,
  • speaker pods are not running,
  • firewall blocks memberlist or BGP.

29.8. HA API Is Not Reachable

bash
getent hosts k8s-api.lab.local
nc -vz k8s-api.lab.local 6443
curl -k https://k8s-api.lab.local:6443/readyz
sudo journalctl -u haproxy -n 100 --no-pager || true
sudo journalctl -u keepalived -n 100 --no-pager || true

Typical causes:

  • VIP is not on the active load balancer,
  • HAProxy backend points to wrong IPs,
  • API server is not running yet,
  • certificate SANs do not include the API name,
  • firewall blocks TCP 6443,
  • DNS points to the wrong IP.

29.9. etcd Problems

bash
kubectl -n kube-system get pods -l component=etcd -o wide
kubectl -n kube-system logs <ETCD_POD>
sudo crictl ps | grep etcd

With local etcdctl on a control-plane node:

bash
sudo ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Typical causes:

  • time drift,
  • disk latency,
  • certificate errors,
  • peer port blocked,
  • quorum lost,
  • wrong advertise URLs.

29.10. MTU Problems

Symptoms:

  • small requests work,
  • larger transfers hang,
  • DNS appears unstable,
  • overlay CNI has packet loss.

Validation:

bash
ip link
tracepath <TARGET_IP>
ping -M do -s 1472 <TARGET_IP>

Reduce CNI MTU only deliberately and consistently. Check underlay, VLANs, bonding, VXLAN/Geneve, and WireGuard/IPsec overhead.

29 / 39

30. Upgrade and Maintenance

30.1. Maintenance Principles

  • Never upgrade without an etcd backup.
  • Do not skip minor versions uncontrolled.
  • Control plane first, then workers.
  • Drain, upgrade, validate, and uncordon one node at a time.
  • Check add-on compatibility.
  • Plan CNI, CSI, and Ingress separately.
  • Control package repositories per Kubernetes minor version.

30.2. etcd Backup

On a control-plane node with stacked etcd:

bash
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-snapshot-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

sudo ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snapshot-*.db -w table

30.3. Drain a Node

bash
kubectl cordon <NODE>
kubectl drain <NODE> --ignore-daemonsets --delete-emptydir-data
# perform maintenance
kubectl uncordon <NODE>

30.4. kubeadm Upgrade Overview

On the first control-plane node:

bash
sudo apt-mark unhold kubeadm || true
sudo apt-get update
sudo apt-get install -y kubeadm=<VERSION>
sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan
sudo kubeadm upgrade apply <VERSION>

Then kubelet/kubectl:

bash
sudo apt-mark unhold kubelet kubectl || true
sudo apt-get install -y kubelet=<VERSION> kubectl=<VERSION>
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

On additional control-plane nodes, you typically use kubeadm upgrade node, then update kubelet/kubectl.

30.5. Check Certificates

bash
sudo kubeadm certs check-expiration

Renew:

bash
sudo kubeadm certs renew all
sudo systemctl restart kubelet

Plan certificate renewal with a maintenance window and backup.

30 / 39

31. Teardown

Teardown is destructive. Back up anything you want to keep first.

31.1. Remove Workloads

bash
kubectl get ns
kubectl delete namespace <NAMESPACE>

31.2. Remove a Node from the Cluster

On the admin side:

bash
kubectl drain <NODE> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <NODE>

On the node:

bash
sudo kubeadm reset
sudo systemctl stop kubelet
sudo systemctl stop containerd || true
sudo systemctl stop crio || true

31.3. Remove CNI and kubeadm Residue

bash
sudo rm -rf /etc/cni/net.d
sudo rm -rf /var/lib/cni
sudo rm -rf /var/lib/kubelet
sudo rm -rf /etc/kubernetes
sudo rm -rf /var/lib/etcd
sudo iptables -F || true
sudo iptables -t nat -F || true
sudo iptables -t mangle -F || true
sudo ipvsadm --clear || true

Caution: broadly flushing iptables/ipvs can affect other services on the host. This is not acceptable on shared hosts.

31.4. Remove Packages

Ubuntu/Debian:

bash
sudo apt-mark unhold kubelet kubeadm kubectl || true
sudo apt-get purge -y kubelet kubeadm kubectl kubernetes-cni || true
sudo apt-get autoremove -y

RHEL/Rocky/Alma:

bash
sudo yum remove -y kubelet kubeadm kubectl kubernetes-cni || true

31.5. k3s/RKE2/k0s/Talos Teardown

k3s:

bash
sudo /usr/local/bin/k3s-uninstall.sh || true
sudo /usr/local/bin/k3s-agent-uninstall.sh || true

RKE2:

bash
sudo /usr/local/bin/rke2-uninstall.sh || true
sudo /usr/local/bin/rke2-agent-uninstall.sh || true

k0s:

bash
sudo k0s stop || true
sudo k0s reset || true

Talos:

bash
talosctl reset --nodes <NODE_IP> --endpoints <ENDPOINT_IP> --graceful=false --reboot
31 / 39

32. Workshop Labs

32.1. Lab 1: Bare-Metal Preflight

Goal: You check whether a physical host is suitable for Kubernetes.

Tasks:

  1. Set hostname.
  2. Check IP, route, and DNS.
  3. Disable swap.
  4. Load kernel modules.
  5. Set sysctl.
  6. Install runtime.
  7. Run crictl info.

Acceptance:

bash
swapon --show
lsmod | grep br_netfilter
sysctl net.ipv4.ip_forward
sudo crictl info

32.2. Lab 2: Single-Node kubeadm

Goal: You build a single Kubernetes node with kubeadm.

Tasks:

  1. Install kubeadm.
  2. Write kubeadm configuration.
  3. Run kubeadm init.
  4. Set kubeconfig.
  5. Remove the control-plane taint.
  6. Install CNI.
  7. Run smoke test.

Acceptance:

bash
kubectl get nodes
kubectl get pods -A
kubectl run test --image=busybox:1.36 --restart=Never -- echo ok

32.3. Lab 3: Multi-Node Join

Goal: You add worker nodes.

Tasks:

  1. Prepare workers.
  2. Generate join command.
  3. Join workers.
  4. Set labels.
  5. Spread a Deployment with three replicas.

Acceptance:

bash
kubectl get nodes -o wide
kubectl create deployment spread --image=nginx:stable --replicas=3
kubectl get pods -o wide

32.4. Lab 4: MetalLB and Ingress

Goal: You provide Services with external IPs.

Tasks:

  1. Install MetalLB.
  2. Create IPAddressPool.
  3. Test LoadBalancer Service.
  4. Install Ingress Controller.
  5. Create example Ingress.

Acceptance:

bash
kubectl get svc -A | grep LoadBalancer
curl http://<EXTERNAL_IP>

32.5. Lab 5: Storage

Goal: You validate dynamic or local storage provisioning.

Tasks:

  1. Check StorageClass.
  2. Create PVC.
  3. Start pod with volume.
  4. Write file.
  5. Delete pod and check behavior.

Acceptance:

bash
kubectl get pvc,pv
kubectl exec storage-smoke -- cat /data/test.txt

32.6. Lab 6: HA Expansion

Goal: You expand a cluster to multiple control-plane nodes.

Tasks:

  1. Provide API endpoint.
  2. Run kubeadm init --upload-certs.
  3. Join additional control-plane nodes.
  4. Redistribute CoreDNS.
  5. Simulate failure of one control-plane node.

Acceptance:

bash
kubectl get nodes
kubectl -n kube-system get pods -l component=kube-apiserver -o wide
kubectl get --raw='/readyz?verbose'
32 / 39

33. Production-Readiness Checklist

A bare-metal Kubernetes cluster is production-ready only when these points are satisfied.

33.1. Architecture

  • Target architecture documented.
  • Node roles documented.
  • Network and CIDR plan documented.
  • API endpoint stable.
  • CNI and CSI decision documented.
  • LoadBalancer model documented.
  • Ingress/Gateway model documented.

33.2. Reproducibility

  • Operating system installation is automated or documented.
  • Package sources are versioned.
  • Kubernetes minor version is pinned.
  • kubeadm configuration is in Git.
  • Add-ons are installed declaratively.
  • Manual steps are justified.

33.3. Security

  • SSH hardened.
  • BMC network separated.
  • RBAC least privilege.
  • admin.conf not shared.
  • Pod Security labels set.
  • Secret encryption evaluated.
  • NetworkPolicies defined.
  • Registry and image policy evaluated.

33.4. Operations

  • etcd backups exist.
  • Restore tested.
  • Upgrade process documented.
  • Monitoring installed.
  • Logs accessible.
  • Alerting defined.
  • Capacity observable.
  • Teardown documented.

33.5. Validation

  • Node health checked.
  • CNI checked.
  • DNS checked.
  • Service routing checked.
  • LoadBalancer checked.
  • Ingress/Gateway checked.
  • Storage checked.
  • Security baseline checked.
33 / 39

34. Common Anti-Patterns

Anti-pattern Why it is problematic Better
kubeadm init without a CIDR plan later change is difficult define CIDRs beforehand
declaring a single control plane “production” control plane is SPOF HA or explicit lab label
installing CNI by gut feeling pod network may not work check CNI requirements first
expecting Docker Engine as runtime Kubernetes uses CRI use containerd or CRI-O
expecting LoadBalancer Services without MetalLB/kube-vip bare metal does not provide this automatically install LB component
distributing admin.conf uncontrolled cluster-admin access users/RBAC/OIDC
using hostPath storage in production insecure and node-bound deliberately choose CSI/Local PV
disabling firewall completely and forgetting it security gap set documented rules
air-gap without a complete image list installation fails halfway through mirror images first
upgrade without etcd backup high recovery risk backup and restore test
34 / 39

35. Provisioning Command Reference

35.1. Preflight

bash
hostnamectl
ip addr
ip route
getent hosts <NAME>
timedatectl
free -h
swapon --show
lsblk
findmnt
sudo cat /sys/class/dmi/id/product_uuid || true

35.2. Runtime

bash
sudo systemctl status containerd --no-pager
sudo systemctl restart containerd
sudo crictl info
sudo crictl ps -a
sudo crictl logs <CONTAINER_ID>
sudo ctr namespaces list

35.3. kubeadm

bash
kubeadm version
sudo kubeadm config images list
sudo kubeadm config images pull
sudo kubeadm init --config kubeadm.yaml --upload-certs
kubeadm token create --print-join-command
sudo kubeadm join <ENDPOINT>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>
sudo kubeadm reset
sudo kubeadm certs check-expiration

35.4. kubectl Basics

bash
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl describe node <NODE>
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get --raw='/readyz?verbose'

35.5. Networking

bash
kubectl get svc,endpoints,endpointslices -A
kubectl get ingress -A
kubectl get gatewayclass,gateway,httproute -A 2>/dev/null || true
kubectl get networkpolicy -A
ss -tulpn
nc -vz <HOST> <PORT>
tracepath <IP>

35.6. Storage

bash
kubectl get storageclass
kubectl get pv,pvc -A
kubectl describe pvc <PVC> -n <NAMESPACE>
kubectl get volumeattachments -A 2>/dev/null || true
lsblk
findmnt

35.7. Logs

bash
sudo journalctl -u kubelet -f
sudo journalctl -u containerd -f
sudo journalctl -u crio -f
kubectl logs <POD> -n <NAMESPACE>
kubectl logs <POD> -n <NAMESPACE> --previous
kubectl logs deployment/<DEPLOYMENT> -n <NAMESPACE> --all-containers
35 / 39
text
kubernetes-workshop/
  docs/
    environments/
      environment-native-baremetal-kubernetes-EN.md
    architecture-kubernetes-EN.md
    concepts-kubernetes-EN.md
    commands-kubernetes-EN.md
    best-practices-kubernetes-EN.md
  ansible/
    inventory.ini
    group_vars/
    playbooks/
    roles/
  clusters/
    bm-single/
      kubeadm.yaml
      cni/
      metallb/
      ingress/
      storage/
    bm-ha/
      kubeadm-ha.yaml
      loadbalancer/
      addons/
  scripts/
    preflight-baremetal.sh
    validate-cluster.sh
    reset-node.sh
  examples/
    smoke/
    network/
    storage/
    security/
36 / 39

37. Example Scripts

37.1. preflight-baremetal.sh

bash
#!/usr/bin/env bash
set -euo pipefail

echo "== Host =="
hostnamectl || true
uname -a

echo "== Network =="
ip addr
ip route

echo "== Time =="
timedatectl || true

echo "== Swap =="
swapon --show || true
free -h

echo "== Kernel modules =="
lsmod | grep -E 'overlay|br_netfilter' || true

echo "== sysctl =="
sysctl net.bridge.bridge-nf-call-iptables || true
sysctl net.ipv4.ip_forward || true

echo "== Runtime =="
systemctl is-active containerd || true
systemctl is-active crio || true
crictl info || true

37.2. validate-cluster.sh

bash
#!/usr/bin/env bash
set -euo pipefail

kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get --raw='/readyz?verbose'

kubectl run validation-dns --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/validation-dns --timeout=120s
kubectl exec validation-dns -- nslookup kubernetes.default.svc.cluster.local
kubectl delete pod validation-dns

37.3. reset-node.sh

bash
#!/usr/bin/env bash
set -euo pipefail

read -r -p "This operation deletes Kubernetes state on this node. Continue? [yes/NO] " answer
if [[ "${answer}" != "yes" ]]; then
  echo "Aborted."
  exit 1
fi

sudo kubeadm reset -f || true
sudo systemctl stop kubelet || true
sudo systemctl stop containerd || true
sudo systemctl stop crio || true
sudo rm -rf /etc/cni/net.d /var/lib/cni /var/lib/kubelet /etc/kubernetes
sudo rm -rf /var/lib/etcd
37 / 39

38. Sources and Further References

Primary references:

  • Kubernetes Documentation: Getting started — https://kubernetes.io/docs/setup/
  • Kubernetes Documentation: Installing kubeadm — https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
  • Kubernetes Documentation: Creating a cluster with kubeadm — https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
  • Kubernetes Documentation: Creating Highly Available Clusters with kubeadm — https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
  • Kubernetes Documentation: Container Runtimes — https://kubernetes.io/docs/setup/production-environment/container-runtimes/
  • MetalLB Documentation — https://metallb.io/
  • Cilium Documentation — https://docs.cilium.io/
  • k3s Documentation — https://docs.k3s.io/
  • RKE2 Documentation — https://docs.rke2.io/
  • Talos Documentation — https://docs.siderolabs.com/talos/
  • k0s Documentation — https://docs.k0sproject.io/
38 / 39

39. Closing

Bare-metal Kubernetes is manageable when you treat it as a chain of controlled decisions:

  1. Prepare hardware and firmware.
  2. Plan network, CIDRs, DNS, VIPs, and ports.
  3. Install the operating system reproducibly.
  4. Configure the runtime correctly.
  5. Execute Kubernetes bootstrap deterministically.
  6. Select CNI, LoadBalancer, Ingress, storage, and add-ons deliberately.
  7. Automate validation.
  8. Document backup, upgrade, security, and teardown.

If you work through this chain cleanly, bare metal is not “magic”; it is a transparent and well-controlled Kubernetes platform.

39 / 39

K8S · Orthevs Academy

31 modules ως ενιαία διαδρομή μάθησης.

Η σειρά παραμένει αμετάβλητη: από τη βασική δομή του lab μέσω τεχνικών θεμελίων και λειτουργικών μοντέλων έως το τελικό troubleshooting. Τα τεχνικά ονόματα καταλόγων διατηρούνται σκόπιμα.

Πρακτικά modules

Το Orthevs ως εφαρμογή

Εγκατάσταση του Orthevs

Η web app του Orthevs μπορεί να εγκατασταθεί σε υποστηριζόμενες κινητές συσκευές και browsers υπολογιστή.

iPhone και iPad

Ανοίξτε το Orthevs στο Safari. Πατήστε Κοινή χρήση και μετά Προσθήκη στην οθόνη αφετηρίας. Ενεργοποιήστε το Άνοιγμα ως web app, αν εμφανίζεται, και πατήστε Προσθήκη.

Android

Για τη web app, ανοίξτε το μενού του browser και επιλέξτε Εγκατάσταση εφαρμογής ή Προσθήκη στην αρχική οθόνη. Επιβεβαιώστε την εγκατάσταση.

Browser υπολογιστή

Ανοίξτε το μενού του browser ή το εικονίδιο εγκατάστασης στη γραμμή διευθύνσεων και επιλέξτε Εγκατάσταση εφαρμογής ή Προσθήκη στο Dock.

Η εγκατάσταση της web app εξαρτάται από τον browser. Αν δεν εμφανιστεί διάλογος εγκατάστασης, ακολουθήστε τα βήματα του browser που αναφέρονται παραπάνω.