1. Purpose

This file documents how to provision Kubernetes in virtual machines for the Kubernetes workshop. It covers local VMs with VirtualBox or VMware, remote VMs rented from a provider, Vagrant-based VM labs, and lightweight alternatives such as kind and minikube.

Use this document when you need a reproducible lab environment where real system services, network paths, the container runtime, kubelet, CNI, storage, ingress, and security controls can be inspected and understood.

This file replaces a short Ubuntu-VM note with a complete provisioning reference. The language is intentionally practical: after each section you should know which option to choose, which commands are required, how to validate the result, and which failure modes are typical.

2. Target State

A VM-based Kubernetes environment should meet three goals:

1. Reproducibility: You can delete the environment and rebuild it with the same parameters.
2. Operational realism: You work with Linux, systemd, a container runtime, kubelet, network adapters, firewalls, certificates, and real cluster components.
3. Security: You separate lab, host, and production identities. You do not use real customer data, production tokens, or unreviewed cluster access.

flowchart TD
    Host[Your host: Linux, macOS, or Windows] --> LocalHypervisor[Local hypervisor]
    Host --> RemoteVM[Rented remote VM]
    LocalHypervisor --> VirtualBox[VirtualBox]
    LocalHypervisor --> VMware[VMware Workstation or Fusion]
    LocalHypervisor --> Vagrant[Vagrant as automation layer]
    VirtualBox --> KubeadmLocal[kubeadm cluster]
    VMware --> KubeadmLocal
    Vagrant --> KubeadmLocal
    Host --> Kind[kind: Kubernetes in containers]
    Host --> Minikube[minikube: local or remote VM]
    RemoteVM --> KubeadmRemote[kubeadm, k3s, RKE2, k0s, or minikube SSH]
    KubeadmLocal --> Labs[Workshop labs]
    Kind --> Labs
    Minikube --> Labs
    KubeadmRemote --> Labs

For fundamentals, short exercises, and CI-like tests, kind or minikube is usually sufficient. For cluster lifecycle, kubelet, kubeadm, CNI, storage, node maintenance, firewalls, and HA, you need real VMs.

3. Scope and Boundaries

This document describes VM-based lab and training environments. It does not replace:

• a production platform architecture,
• a complete cloud IaC reference,
• security hardening according to your organization’s standard,
• provider-specific managed Kubernetes documentation,
• a full disaster-recovery concept.

Even so, production-oriented patterns are used. Remote VMs in particular must be treated as exposed infrastructure: firewalls, SSH hardening, controlled API access, and disciplined teardown are part of the baseline.

4. Decision Model

Choose the provisioning path based on the learning objective.

flowchart TD
    Start[What do you want to learn?] --> Objects[Kubernetes objects]
    Start --> NodeOps[Node and cluster operations]
    Start --> Remote[Remote operations]
    Start --> Repeat[Reproducible local VMs]
    Objects --> Fast[Choose kind or minikube]
    NodeOps --> Kubeadm[Choose kubeadm in VMs]
    Remote --> Provider[Choose rented VMs with a private network]
    Repeat --> Vagrant[Choose Vagrant with VirtualBox or VMware]
    Kubeadm --> HA{Need HA?}
    HA -->|No| SingleOrMulti[Single-node or 1+2]
    HA -->|Yes| ThreeCP[3 control planes plus workers]

5. Reference Architectures

5.1 Local Single-Node Cluster

This option is suitable for fundamentals, manifest exercises, kubectl, probes, services, and first ingress labs.

flowchart TD
    Laptop[Laptop or workstation] --> VM[Ubuntu VM]
    VM --> Runtime[containerd]
    VM --> Kubelet[kubelet]
    VM --> ControlPlane[API Server, Scheduler, Controller Manager, etcd]
    VM --> Pods[Workshop pods]
    VM --> CNI[CNI]
    Laptop --> Kubectl[kubectl]
    Kubectl --> ControlPlane

5.2 Local Multi-Node Cluster

This option is better for scheduling, node maintenance, CNI, NetworkPolicies, storage, and rollout behavior.

flowchart LR
    Host[Host] --> CP[VM cp1: control plane]
    Host --> W1[VM worker1]
    Host --> W2[VM worker2]
    CP --> ETCD[etcd]
    CP --> API[API Server]
    W1 --> Pods1[Pods]
    W2 --> Pods2[Pods]
    API --> W1
    API --> W2
    W1 <-->|Pod network| W2

5.3 Remote VM Cluster

This option is useful for more realistic network and firewall exercises.

flowchart TD
    Admin[Your machine] -->|SSH 22| BastionOrCP[Bastion or control plane]
    Admin -->|kubectl 6443, restricted| API[API Server]
    subgraph Provider[Provider network]
        API --> CP1[Control-plane VM]
        CP1 --> PrivateNet[Private network]
        PrivateNet --> W1[Worker VM]
        PrivateNet --> W2[Worker VM]
        W1 --> Workloads[Workloads]
        W2 --> Workloads
    end
    Internet[Internet] -->|80/443 optional| Ingress[Ingress or LoadBalancer]
    Ingress --> Workloads

5.4 kind Cluster

kind uses containers as Kubernetes nodes. It is not a VM, but it belongs in this file because it is often the most pragmatic workshop environment.

flowchart TD
    Host[Host with Docker or Podman] --> DockerNet[Container network]
    DockerNet --> KindCP[Container: control-plane]
    DockerNet --> KindW1[Container: worker]
    DockerNet --> KindW2[Container: worker]
    KindCP --> APIServer[API Server]
    KindW1 --> Pods1[Pods]
    KindW2 --> Pods2[Pods]
    Host --> Kubectl[kubectl context kind-workshop]
    Kubectl --> APIServer

5.5 minikube Cluster

minikube can run as a VM, container, bare-metal, or SSH-based environment. For this workshop, the Docker, VirtualBox, VMware, and SSH drivers are relevant.

flowchart LR
    Host[Host] --> Profile[minikube profile]
    Profile --> Driver{Driver}
    Driver --> Docker[Docker or Podman]
    Driver --> VM[VirtualBox, VMware, KVM, QEMU]
    Driver --> SSH[Remote VM via SSH]
    Docker --> Cluster[Kubernetes]
    VM --> Cluster
    SSH --> Cluster
    Cluster --> Addons[Ingress, Dashboard, Metrics Server, Registry]

6. Terms and Roles

7. Minimum Requirements

7.1 Host Requirements

7.2 VM Requirements

7.3 Operating System

Recommended:

• Ubuntu Server LTS 24.04 or newer for standard labs,
• Debian Stable for lean labs,
• Rocky Linux, AlmaLinux, or RHEL for Red Hat-oriented environments,
• Talos Linux when demonstrating a Kubernetes-specific immutable OS.

For workshop standardization, Ubuntu Server LTS is the simplest choice because package names, cloud images, and Vagrant boxes are widely available.

8. Planning Checklist

Before creating VMs, clarify these points:

• Which cluster variant do you need: kind, minikube, kubeadm, k3s, RKE2, k0s, or Talos?
• How many nodes do you need?
• Which IP addresses will control-plane and worker nodes use?
• Which API endpoint will be used in certificates and kubeconfig?
• Which pod and service CIDRs will you use?
• Which networks are already used by the host, VPN, office LAN, or provider network?
• Do you need ingress on 80/443?
• Do you need LoadBalancer Services?
• Do you need a local registry or image mirror?
• Which add-ons will be installed?
• How will you tear the environment down?
• Where are kubeconfigs, SSH keys, and temporary tokens stored?
• Are production secrets excluded?

Recommended defaults for labs:

9. Network Design for VM Clusters

VM networking is the most common source of broken labs. Separate these concerns deliberately:

1. Management access: SSH, kubectl, host-to-VM access.
2. Node-to-node communication: kubelet, CNI, pod traffic, control-plane communication.
3. Internet egress: package installation and image pulls.
4. External workload reachability: Ingress, NodePort, LoadBalancer.

flowchart TD
    Host[Host] -->|SSH and kubectl| Mgmt[Management network]
    subgraph Cluster[VM cluster]
        Mgmt --> CP[cp1]
        Mgmt --> W1[worker1]
        Mgmt --> W2[worker2]
        CP <-->|Cluster communication| W1
        CP <-->|Cluster communication| W2
        W1 <-->|Pod traffic| W2
    end
    Cluster -->|Image pulls and packages| Internet[Internet]
    Client[Browser or curl] -->|80/443 or NodePort| Ingress[Ingress/NodePort]
    Ingress --> W1
    Ingress --> W2

9.1 NAT

NAT is useful for internet access from the VM. NAT alone is poor for multi-node Kubernetes because VMs do not automatically reach each other through stable internal addresses.

Use NAT for package downloads, but not as the only cluster network.

9.2 Host-only

Host-only networking is ideal for local labs. Your host and the VMs communicate on an isolated network. The office LAN remains unaffected.

Typical setup:

• Adapter 1: NAT for internet access.
• Adapter 2: Host-only for cluster communication.

9.3 Bridged

Bridged networking attaches VMs directly to the physical LAN. This is practical when other devices on the network should reach Services. It depends on Wi-Fi behavior, DHCP, corporate policies, and network segmentation.

Use bridged networking only when you understand the impact.

9.4 Remote Provider Network

For rented VMs, you need a private network between nodes. Without a private network, node-to-node traffic uses public IPs. That can work in labs, but it is operationally messy and insecure.

Recommendation:

• Public IP only for SSH, API access, and optional ingress.
• Private IP for kubelet, etcd, CNI, and node-to-node communication.
• Configure firewall/security groups restrictively.

9.5 Ports

For kubeadm-like clusters you typically need:

Do not expose all Kubernetes ports to the internet. In remote environments, allow 6443 only from your admin IP or through VPN/bastion.

10. Host Preparation

10.1 Common Host Checks

uname -a
whoami
id
free -h
df -h
ip addr || ifconfig

Also check:

• Hardware virtualization is enabled in BIOS/UEFI.
• On Windows, Hyper-V, WSL2, VirtualBox, and VMware are not in uncontrolled conflict.
• On macOS with Apple Silicon, use ARM-compatible images or tools.
• VPN networks do not collide with pod, service, or host-only CIDRs.
• There is enough free storage for VM disks and container images.

10.2 Linux Host

sudo apt update
sudo apt install -y git curl ca-certificates gnupg lsb-release jq yq make openssh-client

Optional for local container clusters:

sudo apt install -y docker.io
sudo usermod -aG docker "$USER"
newgrp docker

Alternatively, use Podman when your distribution and toolchain support it cleanly.

10.3 macOS Host

With Homebrew:

brew install git curl jq yq kubectl helm kind minikube vagrant

Install VirtualBox or VMware separately. Pay attention to CPU architecture: not every x86 image runs on Apple Silicon.

10.4 Windows Host

Recommended:

• Windows Terminal,
• OpenSSH Client,
• Git,
• kubectl,
• WSL2 for a Linux-like shell,
• VirtualBox, VMware Workstation, or Docker Desktop depending on lab goals.

Avoid unmanaged driver conflicts between Hyper-V, VirtualBox, VMware, and Docker Desktop. For a workshop, standardize one option rather than requiring several hypervisors at once.

11. VM Baseline Standard

Every workshop VM should have reproducible baseline properties.

11.1 Hostname and Name Resolution

sudo hostnamectl set-hostname cp1
hostnamectl

Example /etc/hosts for local labs:

192.168.56.10 cp1
192.168.56.21 worker1
192.168.56.22 worker2

In remote environments, use private provider IP addresses.

11.2 Users and SSH

sudo adduser workshop
sudo usermod -aG sudo workshop
sudo mkdir -p /home/workshop/.ssh
sudo cp ~/.ssh/authorized_keys /home/workshop/.ssh/authorized_keys
sudo chown -R workshop:workshop /home/workshop/.ssh
sudo chmod 700 /home/workshop/.ssh
sudo chmod 600 /home/workshop/.ssh/authorized_keys

Recommendation:

• SSH only with keys.
• Disable password login.
• Disable root login.
• Restrict SSH access on remote VMs to the admin IP.

11.3 Base Packages

sudo apt update
sudo apt install -y \
  apt-transport-https \
  bash-completion \
  ca-certificates \
  conntrack \
  curl \
  dnsutils \
  git \
  gnupg \
  iproute2 \
  iptables \
  jq \
  lsb-release \
  net-tools \
  nfs-common \
  openssl \
  socat \
  software-properties-common \
  tar \
  tcpdump \
  traceroute \
  vim \
  wget

11.4 Kernel Modules and sysctl

cat <<'EOF' | sudo tee /etc/modules-load.d/kubernetes.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

11.5 Swap

For classic kubeadm labs, disable swap:

sudo swapoff -a
sudo sed -i.bak '/\sswap\s/s/^/#/' /etc/fstab
free -h

11.6 Time Synchronization

timedatectl
sudo timedatectl set-ntp true

Time drift causes certificate, token, and etcd problems. Check time carefully, especially after suspending laptop VMs.

12. Provisioning Paths at a Glance

flowchart LR
    Requirement[Requirement] --> Fast[Very fast]
    Requirement --> Real[Operationally realistic]
    Requirement --> Repeatable[Reproducible]
    Requirement --> Remote[Remote]
    Fast --> Kind[kind]
    Fast --> Minikube[minikube]
    Real --> Kubeadm[kubeadm in VMs]
    Repeatable --> Vagrant[Vagrant]
    Remote --> CloudVM[rented VM]
    CloudVM --> Kubeadm
    CloudVM --> K3s[k3s]
    CloudVM --> RKE2[RKE2]
    CloudVM --> K0s[k0s]

13. Local VM with VirtualBox

VirtualBox is suitable for workshops when you need a free and widely available local VM environment.

13.1 Target Topology

flowchart TD
    Host[Host] --> Nat[NAT adapter]
    Host --> HostOnly[Host-only adapter]
    Nat --> Internet[Internet]
    HostOnly --> CP[cp1 192.168.56.10]
    HostOnly --> W1[worker1 192.168.56.21]
    HostOnly --> W2[worker2 192.168.56.22]
    CP <-->|Cluster network| W1
    CP <-->|Cluster network| W2

13.2 Prepare a VirtualBox Network

Example on a host with VBoxManage:

VBoxManage list hostonlyifs
VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.56.1 --netmask 255.255.255.0
VBoxManage list hostonlyifs

Depending on the VirtualBox version, the host-only network may be managed in the GUI. The exact command is less important than a stable isolated network for the VMs.

13.3 VM Parameters

13.4 Installation

1. Create each VM with Ubuntu Server LTS.
2. Enable two network adapters: NAT and host-only.
3. Install OpenSSH Server.
4. Set hostname and static IP or DHCP reservation.
5. Validate internet access and host-to-VM SSH.

ssh workshop@192.168.56.10
ssh workshop@192.168.56.21
ssh workshop@192.168.56.22

13.5 VirtualBox Pitfalls

14. Local VM with VMware Workstation or Fusion

VMware Workstation and VMware Fusion are suitable for stable local VM labs, especially if VMware is already a corporate standard.

14.1 Network Variants

Recommended lab structure:

• One NAT adapter for package installation.
• One host-only or custom VMnet adapter for Kubernetes-internal communication.

14.2 VM Template

Create a base VM:

• Ubuntu Server LTS,
• OpenSSH Server,
• open-vm-tools,
• optional cloud-init,
• all updates installed,
• snapshot base-clean,
• no kubeadm tokens,
• no production SSH keys except the lab key.

Then clone cp1, worker1, and worker2 from this template.

14.3 open-vm-tools

sudo apt update
sudo apt install -y open-vm-tools
sudo systemctl enable --now vmtoolsd || true

14.4 VMware Pitfalls

15. VMware vSphere or ESXi

For production-oriented labs, you can use vSphere, ESXi, or an enterprise virtualization platform. The workflow is similar, but VM creation happens through templates, Content Library, Terraform, PowerCLI, govc, or manual clones.

15.1 Target State

flowchart TD
    Admin[Admin machine] --> vCenter[vCenter or ESXi]
    vCenter --> Template[Ubuntu template]
    Template --> CP1[cp1]
    Template --> CP2[cp2 optional]
    Template --> CP3[cp3 optional]
    Template --> W1[worker1]
    Template --> W2[worker2]
    CP1 --> Cluster[Kubernetes]
    W1 --> Cluster
    W2 --> Cluster

15.2 Template Standard

The template should contain:

• current OS,
• OpenSSH,
• open-vm-tools,
• cloud-init or a comparable initialization mechanism,
• unique machine ID after cloning,
• no persistent host keys if you use it to create many VMs,
• no kubeconfigs or tokens.

After cloning, check:

hostnamectl
cat /etc/machine-id
ip addr
ip route
systemctl status ssh

If cloned VMs have the same machine ID, regenerate it:

sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo systemd-machine-id-setup
sudo reboot

15.3 Remote Console vs SSH

Use the vSphere console only for bootstrap failures. Normal operations should happen through SSH and GitOps/automation.

16. Rented Remote VM

A rented remote VM is useful when your local machine is too small or you want to model networking and operational reality more closely.

16.1 Variants

16.2 Security Model

Remote means: assume internet exposure.

Minimum rules:

• SSH only from your IP or VPN.
• API Server 6443/TCP only from your IP, VPN, or internal network.
• Never expose etcd publicly.
• Do not expose kubelet 10250/TCP publicly.
• Do not open the NodePort range broadly to the internet.
• Do not use production secrets.
• Delete or hard-disable the environment after the workshop.

flowchart TD
    Internet[Internet] -->|blocked| ETCD[etcd ports]
    AdminIP[Your admin IP] -->|22/TCP| SSH[SSH]
    AdminIP -->|6443/TCP optional| API[Kubernetes API]
    Internet -->|80/443 if needed for demo| Ingress[Ingress]
    subgraph VM[Remote VM or provider network]
        SSH --> Node[Linux node]
        API --> Node
        Ingress --> Workload[Demo workload]
    end

16.3 Remote Single-Node with kubeadm

Use this option when you want to demonstrate kubeadm but only have one VM.

Recommendation:

• VM: 4 vCPU, 8 GB RAM, 60 GB disk.
• OS: Ubuntu Server LTS.
• Firewall: SSH and API only for admin IP, 80/443 optional.
• API endpoint: DNS name or fixed public IP.
• CNI: Cilium, Calico, or Flannel.
• Ingress: NGINX Ingress or Gateway controller.

If you want to access the API by public DNS name, add the name to apiServer.certSANs during kubeadm init.

16.4 Remote Multi-Node with kubeadm

Recommended network:

Use private IPs for --apiserver-advertise-address, kubelet node IP, and node-to-node communication.

# On each node, if several interfaces exist:
echo 'KUBELET_EXTRA_ARGS=--node-ip=<PRIVATE_IP>' | sudo tee /etc/default/kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet || true

16.5 Remote Firewall Example with ufw

On cp1:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from <ADMIN_IP>/32 to any port 22 proto tcp
sudo ufw allow from <ADMIN_IP>/32 to any port 6443 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 6443 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 2379 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 2380 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 10250 proto tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

On workers:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from <ADMIN_IP>/32 to any port 22 proto tcp
sudo ufw allow from 10.10.0.0/24 to any port 10250 proto tcp
sudo ufw allow from 10.10.0.0/24
sudo ufw enable
sudo ufw status verbose

Adjust CNI-specific ports. Cilium, Calico, Flannel, and other plugins have different requirements.

16.6 cloud-init for Remote VMs

Example for minimal preparation. This snippet does not install Kubernetes; it creates a clean baseline.

#cloud-config
package_update: true
package_upgrade: true
users:
  - name: workshop
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-ed25519 AAAA...your-lab-key
packages:
  - ca-certificates
  - curl
  - git
  - gnupg
  - jq
  - openssh-server
  - vim
runcmd:
  - timedatectl set-ntp true
  - swapoff -a
  - sed -i.bak '/\sswap\s/s/^/#/' /etc/fstab
  - modprobe overlay
  - modprobe br_netfilter
  - printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/kubernetes.conf
  - printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n' > /etc/sysctl.d/99-kubernetes.conf
  - sysctl --system

17. Kubernetes with kubeadm in VMs

kubeadm is the standard path when you want to learn Kubernetes bootstrap, real nodes, and cluster lifecycle.

17.1 kubeadm Workflow

sequenceDiagram
    participant Admin as You
    participant Node as VM/Node
    participant Runtime as containerd
    participant Kubeadm as kubeadm
    participant API as API Server
    participant CNI as CNI
    Admin->>Node: Prepare OS
    Admin->>Runtime: Install and configure containerd
    Admin->>Node: Install kubelet, kubeadm, kubectl
    Admin->>Kubeadm: kubeadm init
    Kubeadm->>API: Create control plane
    Admin->>CNI: Install CNI
    Admin->>Kubeadm: kubeadm join on workers
    Admin->>API: Validate cluster

17.2 Common Variables

export K8S_MINOR="v1.36"
export K8S_VERSION="v1.36.1"
export POD_CIDR="10.244.0.0/16"
export SERVICE_CIDR="10.96.0.0/12"
export CONTROL_PLANE_ENDPOINT="192.168.56.10:6443"

Adjust the version before each workshop and pin all nodes to the same minor version.

17.3 Install containerd

Ubuntu/Debian:

sudo apt update
sudo apt install -y containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl enable --now containerd
sudo systemctl restart containerd
sudo systemctl status containerd --no-pager

Validation:

sudo crictl info || true
sudo ctr version

If crictl does not yet find a runtime, create a configuration:

cat <<'EOF' | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

17.4 Install Kubernetes Packages: Ubuntu/Debian

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings

curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /
EOF

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

Validation:

kubeadm version
kubectl version --client
systemctl status kubelet --no-pager || true

Before kubeadm init or kubeadm join, the kubelet service may appear unhealthy. It is still waiting for cluster configuration.

17.5 Install Kubernetes Packages: RHEL/Rocky/Alma

export K8S_MINOR="v1.36"

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/rpm/repodata/repomd.xml.key
EOF

sudo dnf install -y kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

Firewall, SELinux, and runtime configuration depend more strongly on distribution and organizational standards. For workshops, document every deviation explicitly.

17.6 kubeadm Configuration

Example kubeadm-config.yaml for local VMs:

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    - name: node-ip
      value: 192.168.56.10
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.1
controlPlaneEndpoint: 192.168.56.10:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - 192.168.56.10
    - cp1
    - kubernetes.local
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

For remote VMs, use private IPs and only add the public DNS name as a SAN if kubectl really needs to access the API Server directly from outside.

18. Single-Node kubeadm

18.1 When This Option Fits

Single-node kubeadm fits:

• fundamentals,
• API Server, etcd, and kubelet on one machine,
• kubeadm bootstrap,
• ingress and storage fundamentals,
• remote demo VMs,
• small homelab or training environments.

It does not fit HA, real scheduling across multiple nodes, or failure-behavior exercises.

18.2 Initialization

On the VM:

sudo kubeadm init --config kubeadm-config.yaml

Kubeconfig for the lab user:

mkdir -p "$HOME/.kube"
sudo cp /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

18.3 Allow Workloads on the Control-Plane Node

Single-node clusters need workloads on the control-plane node:

kubectl taint nodes --all node-role.kubernetes.io/control-plane- || true
kubectl get nodes -o wide

18.4 Install CNI

Without CNI, the node typically remains NotReady.

Example with a CNI manifest from your repository:

kubectl apply -f addons/cni/flannel.yaml
# or
kubectl apply -f addons/cni/calico.yaml
# or Cilium via Helm/CLI according to workshop standard

Validation:

kubectl get nodes -o wide
kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp | tail -50

18.5 Test Deployment

kubectl create namespace demo
kubectl create deployment web -n demo --image=nginx:stable --replicas=2
kubectl expose deployment web -n demo --port=80 --target-port=80
kubectl get deploy,rs,pod,svc -n demo -o wide
kubectl run curl -n demo --rm -it --image=curlimages/curl --restart=Never -- curl -I http://web.demo.svc.cluster.local

19. Multi-Node kubeadm

19.1 Target Topology

flowchart LR
    CP[cp1 192.168.56.10] --> API[API Server]
    CP --> ETCD[etcd]
    CP --> CM[Controller Manager]
    CP --> SCHED[Scheduler]
    W1[worker1 192.168.56.21] --> Kubelet1[kubelet]
    W2[worker2 192.168.56.22] --> Kubelet2[kubelet]
    API --> Kubelet1
    API --> Kubelet2
    W1 <-->|CNI pod network| W2

19.2 Preparation on All Nodes

• Set unique hostname.
• Check IP addresses.
• Disable swap.
• Set kernel modules and sysctl.
• Install containerd.
• Install kubelet, kubeadm, and kubectl.
• Synchronize time.
• Configure firewalls appropriately.

Validation on each node:

hostname -f || hostname
ip addr
ip route
sudo systemctl is-active containerd
kubeadm version

19.3 Initialize Control Plane

On cp1:

sudo kubeadm init --config kubeadm-config.yaml
mkdir -p "$HOME/.kube"
sudo cp /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

Generate join command:

kubeadm token create --print-join-command

19.4 Join Workers

On worker1 and worker2:

sudo kubeadm join 192.168.56.10:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

If several interfaces exist, set the node IP before joining or restarting kubelet:

echo 'KUBELET_EXTRA_ARGS=--node-ip=192.168.56.21' | sudo tee /etc/default/kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet

19.5 Validation

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl describe node worker1
kubectl get events -A --sort-by=.lastTimestamp | tail -80

Expected state:

• all nodes are Ready,
• CoreDNS runs,
• CNI pods run on all nodes,
• workload pods are distributed across workers.

20. Highly Available kubeadm VM Clusters

HA is useful when you want to demonstrate etcd quorum, control-plane endpoints, certificates, load balancers, and maintenance workflows.

20.1 Topology with Stacked etcd

flowchart TD
    LB[Control Plane Endpoint: 192.168.56.100:6443] --> CP1[cp1]
    LB --> CP2[cp2]
    LB --> CP3[cp3]
    CP1 <-->|etcd peer| CP2
    CP2 <-->|etcd peer| CP3
    CP3 <-->|etcd peer| CP1
    CP1 --> W1[worker1]
    CP2 --> W2[worker2]
    CP3 --> W3[worker3 optional]

20.2 Requirements

• Three control-plane VMs for etcd quorum.
• A stable controlPlaneEndpoint before kubeadm init.
• Load balancer or virtual IP in front of API servers.
• Time synchronized.
• Same Kubernetes minor version on all nodes.
• Private node-to-node communication.

20.3 Load-Balancer Options

20.4 kubeadm HA Init

Example excerpt:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.1
controlPlaneEndpoint: 192.168.56.100:6443
networking:
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
    - 192.168.56.100
    - k8s-api.local

Initialize first control-plane node:

sudo kubeadm init --config kubeadm-ha.yaml --upload-certs

Join additional control-plane nodes:

sudo kubeadm join 192.168.56.100:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane \
  --certificate-key <CERTIFICATE_KEY>

Join workers:

sudo kubeadm join 192.168.56.100:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

20.5 HA Validation

kubectl get nodes -o wide
kubectl -n kube-system get pods -l component=etcd -o wide
kubectl -n kube-system get pods -l component=kube-apiserver -o wide
kubectl get --raw=/readyz?verbose

Controlled test:

sudo systemctl stop kubelet
# on one control-plane node, not on all

Then check whether the API endpoint continues to respond.

21. Kubernetes with kind

kind means Kubernetes in Docker. It is not a VM cluster, but it belongs in this file because it is often the most pragmatic workshop environment.

21.1 When to Use kind

Use kind when you:

• want to start quickly,
• need multi-node behavior without real VMs,
• want to run manifest tests locally or in CI,
• want to switch Kubernetes versions quickly,
• do not want a hypervisor dependency.

Do not use kind as a substitute for:

• kubeadm bootstrap,
• VM network failure modes,
• real node maintenance,
• production operations.

21.2 Installation

Linux AMD64:

curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind version

With Homebrew:

brew install kind
kind version

21.3 Single-Node kind

kind create cluster --name workshop
kubectl cluster-info --context kind-workshop
kubectl get nodes -o wide

Delete:

kind delete cluster --name workshop

21.4 Multi-Node kind with Port Mapping

File kind-workshop.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: workshop
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        protocol: TCP
      - containerPort: 443
        hostPort: 8443
        protocol: TCP
  - role: worker
  - role: worker

Create cluster:

kind create cluster --config kind-workshop.yaml
kubectl get nodes -o wide

21.5 Load Image into kind

docker build -t workshop/app:dev .
kind load docker-image workshop/app:dev --name workshop
kubectl set image deployment/app app=workshop/app:dev -n demo

21.6 Ingress with kind

With simple port mapping, you can map ingress to host ports. Example:

kubectl apply -f addons/ingress-nginx/kind.yaml
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=180s

Then test via localhost:8080 or localhost:8443, depending on your ingress setup.

21.7 Local Registry for kind

A local registry is useful for workshops with custom images.

docker run -d --restart=always -p 127.0.0.1:5001:5000 --name kind-registry registry:2

Then configure kind so the nodes can reach the registry. Use a script or documented workshop template so all participants get identical behavior.

21.8 kind Troubleshooting

22. Kubernetes with minikube

minikube is a local Kubernetes environment with many drivers and add-ons. It is especially useful for beginners because many tasks are simplified through minikube commands.

22.1 When to Use minikube

Use minikube when you want:

• a single comfortable learning environment,
• ingress, dashboard, Metrics Server, or registry through add-ons,
• to compare different drivers,
• to use a remote VM via SSH as the target.

22.2 Installation

Linux AMD64:

curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube version

Homebrew:

brew install minikube
minikube version

22.3 Driver Selection

22.4 minikube with Docker Driver

minikube start \
  --driver=docker \
  --kubernetes-version=v1.36.1 \
  --cpus=4 \
  --memory=8192 \
  --disk-size=40g

kubectl get nodes -o wide
minikube status

22.5 minikube with VirtualBox

minikube start \
  --driver=virtualbox \
  --kubernetes-version=v1.36.1 \
  --cpus=4 \
  --memory=8192 \
  --disk-size=40g

Set as default:

minikube config set driver virtualbox

22.6 minikube with VMware

minikube start \
  --driver=vmware \
  --kubernetes-version=v1.36.1 \
  --cpus=4 \
  --memory=8192 \
  --disk-size=40g

Set as default:

minikube config set driver vmware

22.7 minikube on a Remote VM via SSH

This option is useful if you cannot use local virtualization.

On your machine:

minikube start \
  --driver=ssh \
  --ssh-ip-address=<REMOTE_VM_IP> \
  --kubernetes-version=v1.36.1

Prerequisites on the remote VM:

• Linux with SSH access,
• sufficient CPU, RAM, and disk,
• package installation possible,
• no colliding Kubernetes or Docker remnants,
• firewall allows SSH from your machine.

22.8 Multi-Node minikube

minikube start \
  --driver=docker \
  --nodes=3 \
  --cpus=4 \
  --memory=8192 \
  --kubernetes-version=v1.36.1

kubectl get nodes -o wide

22.9 Add-ons

minikube addons list
minikube addons enable metrics-server
minikube addons enable ingress
minikube addons enable dashboard

Dashboard:

minikube dashboard

Test ingress:

kubectl get pods -n ingress-nginx
kubectl get ingress -A

22.10 LoadBalancer with minikube tunnel

minikube tunnel

Depending on the system, this command requires elevated privileges and remains in the foreground. For workshops, explain that the tunnel is part of the environment and must be stopped.

22.11 Profiles

minikube profile list
minikube start -p workshop-a --driver=docker
minikube start -p workshop-b --driver=docker
minikube profile workshop-a
kubectl config current-context

Profiles prevent participants from accidentally reusing old clusters.

22.12 minikube Teardown

minikube stop
minikube delete
minikube delete --all --purge

23. Vagrant for Local VM Provisioning

Vagrant is an automation layer for reproducible VMs. For workshops, it is useful because a Vagrantfile can define several VMs consistently.

23.1 When Vagrant Fits

Vagrant fits when you want to:

• create local multi-node VMs reproducibly,
• use VirtualBox or VMware as provider,
• build a standardized trainer setup,
• version provisioning scripts,
• destroy VMs quickly after the lab.

Vagrant is less suitable for arbitrary rented remote VMs. For those, cloud-init, Terraform, Ansible, or provider-specific APIs are usually better.

23.2 Installation

vagrant version
vagrant plugin list

VirtualBox is directly supported by Vagrant. For VMware, install the VMware provider and the matching VMware Utility.

Install VMware provider:

vagrant plugin install vagrant-vmware-desktop
vagrant plugin list

23.3 Project Structure

kubernetes-vagrant-lab/
├── Vagrantfile
├── scripts/
│   └── vagrant/
│       ├── bootstrap-common.sh
│       ├── init-control-plane.sh
│       └── join-worker.sh
└── manifests/
    └── cni/
        └── flannel.yaml

23.4 Vagrantfile for 1 Control Plane and 2 Workers

# Kubernetes VM lab: one control plane and two workers.
# Provider: VirtualBox by default, VMware via --provider=vmware_desktop.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/noble64"
  config.vm.synced_folder ".", "/vagrant", type: "virtualbox"

  nodes = [
    { name: "cp1",     ip: "192.168.56.10", role: "control-plane", memory: 4096, cpus: 2 },
    { name: "worker1", ip: "192.168.56.21", role: "worker",        memory: 4096, cpus: 2 },
    { name: "worker2", ip: "192.168.56.22", role: "worker",        memory: 4096, cpus: 2 }
  ]

  nodes.each do |node|
    config.vm.define node[:name] do |machine|
      machine.vm.hostname = node[:name]
      machine.vm.network "private_network", ip: node[:ip]

      machine.vm.provider "virtualbox" do |vb|
        vb.name = "k8s-#{node[:name]}"
        vb.memory = node[:memory]
        vb.cpus = node[:cpus]
      end

      machine.vm.provider "vmware_desktop" do |vmw|
        vmw.vmx["displayName"] = "k8s-#{node[:name]}"
        vmw.vmx["memsize"] = node[:memory].to_s
        vmw.vmx["numvcpus"] = node[:cpus].to_s
      end

      machine.vm.provision "shell", path: "scripts/vagrant/bootstrap-common.sh", args: [node[:ip]]

      if node[:role] == "control-plane"
        machine.vm.provision "shell", path: "scripts/vagrant/init-control-plane.sh", args: [node[:ip]]
      else
        machine.vm.provision "shell", path: "scripts/vagrant/join-worker.sh", args: [node[:ip]]
      end
    end
  end
end

23.5 Common Provisioning Script

File scripts/vagrant/bootstrap-common.sh:

#!/usr/bin/env bash
set -euo pipefail

NODE_IP="${1:?node ip missing}"
K8S_MINOR="v1.36"

export DEBIAN_FRONTEND=noninteractive

sudo hostnamectl set-hostname "$(hostname -s)"

cat <<EOF | sudo tee -a /etc/hosts
192.168.56.10 cp1
192.168.56.21 worker1
192.168.56.22 worker2
EOF

sudo swapoff -a
sudo sed -i.bak '/\sswap\s/s/^/#/' /etc/fstab

cat <<'EOF' | sudo tee /etc/modules-load.d/kubernetes.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg jq containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl enable --now containerd
sudo systemctl restart containerd

sudo mkdir -p /etc/apt/keyrings
curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /
EOF

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

echo "KUBELET_EXTRA_ARGS=--node-ip=${NODE_IP}" | sudo tee /etc/default/kubelet
sudo systemctl enable kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet || true

23.6 Control-Plane Script

File scripts/vagrant/init-control-plane.sh:

#!/usr/bin/env bash
set -euo pipefail

NODE_IP="${1:?node ip missing}"
K8S_VERSION="v1.36.1"

if [ -f /etc/kubernetes/admin.conf ]; then
  echo "Control plane already initialized."
  exit 0
fi

cat <<EOF >/tmp/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: ${NODE_IP}
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    - name: node-ip
      value: ${NODE_IP}
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: ${K8S_VERSION}
controlPlaneEndpoint: ${NODE_IP}:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
    - ${NODE_IP}
    - cp1
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

sudo kubeadm init --config /tmp/kubeadm-config.yaml

mkdir -p /home/vagrant/.kube
sudo cp /etc/kubernetes/admin.conf /home/vagrant/.kube/config
sudo chown -R vagrant:vagrant /home/vagrant/.kube

# Install your standardized CNI manifest here in the workshop.
# Example:
# sudo -u vagrant kubectl apply -f /vagrant/manifests/cni/flannel.yaml

sudo kubeadm token create --print-join-command | sudo tee /vagrant/.join-command >/dev/null
sudo chmod 0644 /vagrant/.join-command

23.7 Worker Join Script

File scripts/vagrant/join-worker.sh:

#!/usr/bin/env bash
set -euo pipefail

if [ -f /etc/kubernetes/kubelet.conf ]; then
  echo "Worker already joined."
  exit 0
fi

for i in $(seq 1 60); do
  if [ -f /vagrant/.join-command ]; then
    break
  fi
  echo "Waiting for join command..."
  sleep 5
done

if [ ! -f /vagrant/.join-command ]; then
  echo "Join command missing."
  exit 1
fi

JOIN_COMMAND="$(cat /vagrant/.join-command)"
sudo ${JOIN_COMMAND}

23.8 Vagrant Commands

VirtualBox:

vagrant up --provider=virtualbox
vagrant status
vagrant ssh cp1

VMware:

vagrant up --provider=vmware_desktop
vagrant status
vagrant ssh cp1

Run provisioning again:

vagrant provision cp1
vagrant provision worker1
vagrant provision worker2

Snapshot:

vagrant snapshot save cp1 before-cni
vagrant snapshot list cp1
vagrant snapshot restore cp1 before-cni

Teardown:

vagrant destroy -f
rm -f .join-command

23.9 Vagrant Pitfalls

24. k3s in VMs

k3s is a lightweight Kubernetes distribution. It is suitable for edge, homelab, fast remote VMs, and workshops where kubeadm is not the learning objective.

24.1 Single-Node k3s

curl -sfL https://get.k3s.io | sh -
sudo kubectl get nodes -o wide
sudo kubectl get pods -A

Kubeconfig for your user:

mkdir -p "$HOME/.kube"
sudo cp /etc/rancher/k3s/k3s.yaml "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"
sed -i "s/127.0.0.1/<NODE_IP_OR_DNS>/" "$HOME/.kube/config"

24.2 k3s Without Bundled Components

If you want to demonstrate your own add-ons:

curl -sfL https://get.k3s.io | sh -s - \
  --disable traefik \
  --disable servicelb \
  --disable local-storage

Then install ingress, load balancer, and storage according to workshop standards.

24.3 k3s Multi-Node

Server:

curl -sfL https://get.k3s.io | sh -s - server \
  --node-ip 10.10.0.10 \
  --tls-san k8s.example.net

sudo cat /var/lib/rancher/k3s/server/node-token

Agent:

curl -sfL https://get.k3s.io | K3S_URL=https://10.10.0.10:6443 K3S_TOKEN=<TOKEN> sh -s - agent \
  --node-ip 10.10.0.21

25. RKE2 in VMs

RKE2 is a Kubernetes distribution with a stronger focus on secure defaults and enterprise operations. It is useful when Rancher/SUSE contexts or hardening-oriented platforms are part of the workshop.

25.1 Single-Server RKE2

curl -sfL https://get.rke2.io | sudo sh -
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
sudo journalctl -u rke2-server -f

Kubeconfig:

mkdir -p "$HOME/.kube"
sudo cp /etc/rancher/rke2/rke2.yaml "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"
export PATH="$PATH:/var/lib/rancher/rke2/bin"
kubectl get nodes -o wide

25.2 RKE2 HA Base Model

RKE2 HA needs a fixed registration address in front of the server nodes and an odd number of server nodes.

flowchart TD
    Reg[Registration address or load balancer] --> S1[server1]
    Reg --> S2[server2]
    Reg --> S3[server3]
    S1 <-->|etcd| S2
    S2 <-->|etcd| S3
    S3 <-->|etcd| S1
    Reg --> A1[agent1]
    Reg --> A2[agent2]

26. k0s in VMs

k0s is a lean Kubernetes distribution with controller and worker roles. It is useful as an alternative workshop variant when comparing different distributions.

26.1 Single-Node k0s

curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
sudo k0s install controller --single
sudo k0s start
sudo k0s status
sudo k0s kubectl get nodes -o wide

Kubeconfig:

mkdir -p "$HOME/.kube"
sudo k0s kubeconfig admin > "$HOME/.kube/config"
chmod 600 "$HOME/.kube/config"
kubectl get nodes -o wide

26.2 Worker Join

On the controller:

sudo k0s token create --role=worker > worker-token

On the worker:

sudo k0s install worker --token-file worker-token
sudo k0s start

27. Talos Linux in VMs

Talos Linux is a specialized immutable operating system for Kubernetes. You do not manage nodes through classic SSH, but through talosctl and declarative machine configuration.

27.1 When Talos Fits

Talos fits when you want to demonstrate:

• Kubernetes hosts without an SSH operating model,
• API-driven node configuration,
• immutable infrastructure,
• clear separation between OS configuration and Kubernetes workloads.

Talos is not the first path for Kubernetes beginners because it teaches Kubernetes and a new OS operating model at the same time.

27.2 Talos in VirtualBox or VMware

Workflow:

1. Download the Talos image or ISO suitable for the platform.
2. Create VM with sufficient CPU, RAM, and disk.
3. Configure stable networking.
4. Generate machine configuration with talosctl gen config.
5. Apply configuration to control plane and workers.
6. Bootstrap the cluster.
7. Retrieve kubeconfig.

Schematic commands:

talosctl gen config workshop https://192.168.56.10:6443

talosctl apply-config --insecure --nodes 192.168.56.10 --file controlplane.yaml
talosctl apply-config --insecure --nodes 192.168.56.21 --file worker.yaml

talosctl bootstrap --nodes 192.168.56.10 --endpoints 192.168.56.10
talosctl kubeconfig --nodes 192.168.56.10 --endpoints 192.168.56.10
kubectl get nodes -o wide

Check the official Talos platform documentation for VirtualBox, VMware, and vSphere because image types, guestinfo, disk options, and platform details differ.

28. Add-ons After Cluster Creation

After bootstrap, a Kubernetes cluster is not yet a full platform. For workshops, install add-ons deliberately and in a fixed order.

flowchart TD
    Bootstrap[kubeadm, kind, minikube, or distribution] --> CNI[CNI]
    CNI --> DNS[Validate CoreDNS]
    DNS --> Metrics[Metrics Server]
    Metrics --> Ingress[Ingress or Gateway]
    Ingress --> Certs[cert-manager optional]
    Certs --> Storage[StorageClass]
    Storage --> Observability[Monitoring, logging]
    Observability --> GitOps[Argo CD or Flux optional]

28.1 Recommended Order

1. Install CNI.
2. Validate nodes and CoreDNS.
3. Install Metrics Server.
4. Provide a StorageClass.
5. Install Ingress Controller or Gateway Controller.
6. Install certificate management if TLS is part of the lab.
7. Install observability.
8. Optionally add GitOps.
9. Activate security policies.

28.2 Add-on Matrix

29. Storage in VM Environments

Storage in VMs is often deceptive: a virtual disk is not automatically highly available storage. Distinguish lab persistence from production-like storage.

29.1 Options

29.2 Local Path Provisioner for Labs

If your cluster has no StorageClass, install a simple local StorageClass only for labs.

Validation:

kubectl get storageclass
kubectl create namespace storage-test
cat <<'EOF' | kubectl apply -n storage-test -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF
kubectl get pvc -n storage-test
kubectl describe pvc data -n storage-test

29.3 Snapshot Warning

VM snapshots do not replace Kubernetes backups. A snapshot of running control-plane VMs can make etcd inconsistent if taken without coordination. For etcd, use etcdctl snapshot save or distribution-specific backup mechanisms.

30. Ingress, Gateway, and Load Balancing

30.1 NodePort

NodePort is simple, but often inconvenient for workshop access.

kubectl expose deployment web -n demo --type=NodePort --port=80
kubectl get svc -n demo web -o wide

30.2 Ingress

Ingress always requires an Ingress Controller.

kubectl get ingressclass
kubectl get pods -A | grep -i ingress || true

Example resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: demo
spec:
  ingressClassName: nginx
  rules:
    - host: web.localtest.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

30.3 Gateway API

Gateway API is suitable for advanced labs because it separates roles and responsibilities more cleanly.

flowchart TD
    Platform[Platform team] --> GatewayClass[GatewayClass]
    Platform --> Gateway[Gateway]
    AppTeam[App team] --> HTTPRoute[HTTPRoute]
    HTTPRoute --> Service[Service]
    Service --> Pods[Pods]
    Gateway --> HTTPRoute

30.4 MetalLB in VM Networks

MetalLB gives Services of type LoadBalancer an IP from a pool you define. In local host-only networks, choose a free range, for example 192.168.56.200-192.168.56.220.

Conceptual example:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.56.200-192.168.56.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - lab-pool

First verify that DHCP does not allocate these addresses.

31. Security Baseline

31.1 VM Security

• Use dedicated lab SSH keys.
• Disable password login where possible.
• Open remote firewalls minimally.
• Separate lab networks from production networks.
• Do not store production secrets in VMs.
• Delete kubeadm tokens after use or keep them short-lived.
• Remove kubeconfigs after teardown.

31.2 Kubernetes Security

After cluster setup:

kubectl get clusterrolebinding
kubectl get serviceaccount -A
kubectl auth can-i '*' '*' --all-namespaces
kubectl get ns --show-labels

Namespace baseline:

kubectl create namespace demo
kubectl label namespace demo pod-security.kubernetes.io/enforce=baseline
kubectl label namespace demo pod-security.kubernetes.io/audit=restricted
kubectl label namespace demo pod-security.kubernetes.io/warn=restricted

31.3 Secret Rules

• No real passwords in Git.
• No production registry tokens.
• No cloud credentials in workshop namespaces.
• Delete Secrets after exercises.
• Do not treat base64 as encryption.

kubectl get secrets -A
kubectl delete namespace demo

32. Observability and Validation

32.1 Baseline Validation

kubectl version
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl get componentstatuses 2>/dev/null || true
kubectl get --raw='/readyz?verbose'

32.2 Node View

systemctl status kubelet --no-pager
systemctl status containerd --no-pager
journalctl -u kubelet -n 200 --no-pager
journalctl -u containerd -n 100 --no-pager
sudo crictl ps -a
sudo crictl images

32.3 Network View

ip addr
ip route
ss -tulpen
sudo iptables-save | head -80
kubectl get svc,endpoints,endpointslices -A
kubectl run net-debug --rm -it --image=busybox:1.36 --restart=Never -- sh

Inside the debug pod:

nslookup kubernetes.default.svc.cluster.local
wget -qO- http://kubernetes.default.svc.cluster.local

32.4 Metrics Server

kubectl top nodes
kubectl top pods -A

If kubectl top does not work, first check Metrics Server, kubelet certificates, and APIService status:

kubectl get apiservice | grep metrics
kubectl -n kube-system logs deployment/metrics-server

33. Upgrades, Maintenance, and Snapshots

33.1 Principles

• Do not upgrade during a workshop without a time window.
• Do not mix minor versions randomly.
• Upgrade control plane first, then workers.
• Take an etcd backup before kubeadm control-plane upgrades.
• Afterward check nodes, pods, CoreDNS, CNI, and add-ons.

33.2 Rough kubeadm Upgrade Flow

On control plane:

sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm=<TARGET_VERSION>
sudo apt-mark hold kubeadm
kubeadm version
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply <TARGET_VERSION>

Then kubelet/kubectl:

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=<TARGET_VERSION> kubectl=<TARGET_VERSION>
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

33.3 VM Snapshots

Snapshots are useful for labs:

• snapshot before kubeadm init,
• snapshot after OS baseline,
• snapshot after successful cluster bootstrap,
• snapshot before risky exercises.

But:

• Do not present uncoordinated snapshots of all etcd members as backups.
• After snapshot restore, check time and certificates.
• For multi-node clusters, consider consistent snapshot timing.

34. Teardown

34.1 Tear Down kubeadm Cluster

On workers:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/cni /var/lib/kubelet /etc/kubernetes
sudo systemctl restart containerd

On control plane:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d /var/lib/cni /var/lib/kubelet /etc/kubernetes
rm -rf "$HOME/.kube"
sudo systemctl restart containerd

Only clean iptables in a controlled way if the VM is fully dedicated to the lab:

sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X

34.2 kind

kind get clusters
kind delete cluster --name workshop
docker ps -a | grep kind || true

34.3 minikube

minikube profile list
minikube stop
minikube delete --all --purge

34.4 Vagrant

vagrant status
vagrant destroy -f
rm -f .join-command

34.5 Remote VMs

• Delete local kubeconfig.
• Remove SSH key from authorized_keys if the VM remains.
• Reset provider firewall.
• Delete VM if it existed only for the workshop.
• Remove DNS records.
• Delete volumes, snapshots, and floating IPs to avoid ongoing cost.

35. Troubleshooting Catalog

35.1 kubeadm preflight Fails

Diagnosis:

sudo kubeadm init --config kubeadm-config.yaml --v=5
sudo journalctl -u kubelet -n 200 --no-pager
sudo crictl info

Common causes:

• swap enabled,
• containerd not running,
• wrong cgroup driver,
• ports already in use,
• old Kubernetes files present,
• hostname or IP not unique,
• firewall blocks API or kubelet.

35.2 Node Remains NotReady

kubectl describe node <node>
kubectl get pods -n kube-system -o wide
journalctl -u kubelet -n 200 --no-pager

Typical causes:

• CNI missing,
• CNI pods crash,
• node IP wrong,
• pod CIDR collides,
• kubelet cannot reach runtime.

35.3 CoreDNS Remains Pending or CrashLoopBackOff

kubectl -n kube-system describe deploy coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl describe node

Causes:

• node not Ready,
• CNI missing,
• insufficient resources,
• control-plane taint not removed in a single-node cluster.

35.4 Worker Cannot Join

sudo kubeadm join ... --v=5
nc -vz 192.168.56.10 6443
curl -k https://192.168.56.10:6443/readyz

Causes:

• token expired,
• CA hash wrong,
• API endpoint not reachable,
• firewall blocks traffic,
• wrong interface,
• certificate SAN does not match.

35.5 kubectl Reports Certificate Errors

kubectl config view --minify
openssl s_client -connect <api-endpoint>:6443 -showcerts </dev/null

Causes:

• API is addressed through a DNS name/IP that is not in the certificate,
• kubeconfig points to old IP,
• cluster was restored from snapshot with different network configuration.

35.6 Pods Stuck in ImagePullBackOff

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
sudo crictl pull <image>

Causes:

• image name wrong,
• registry unreachable,
• authentication missing,
• local images not loaded into kind/minikube,
• remote VM has no egress.

35.7 Service Has No Endpoints

kubectl get svc,endpoints,endpointslices -n demo
kubectl get pods -n demo --show-labels
kubectl describe svc <service> -n demo

Causes:

• selector does not match pod labels,
• pods not Ready,
• wrong namespace,
• Deployment not rolled out.

35.8 Ingress Is Not Reachable

kubectl get ingressclass
kubectl get ingress -A
kubectl get pods -A | grep -i ingress
kubectl describe ingress <name> -n <namespace>

Causes:

• Ingress Controller missing,
• wrong ingressClassName,
• DNS/Host header missing,
• port mapping missing in kind,
• minikube tunnel not running,
• firewall blocks 80/443.

35.9 Vagrant Provisioning Fails

vagrant status
vagrant ssh cp1
vagrant provision cp1
vagrant provision worker1

Causes:

• synced folder not mounted,
• scripts not executable,
• provider plugin missing,
• VM has old kubeadm state,
• join file not generated,
• VM host-only network missing.

35.10 Remote VM Becomes Unreachable After Firewall Rule

Prevention:

• Open a new SSH session before changing firewall rules.
• Use provider console as emergency access.
• Explicitly allow your admin IP.
• Enable firewall only after verification.

Diagnosis through provider console:

sudo ufw status verbose
sudo iptables -S
sudo ss -tulpen | grep ssh

36. Workshop Labs

Lab 1: kind in 10 Minutes

Goal: Create a local multi-node cluster without a VM.

Tasks:

1. Install kind.
2. Write multi-node configuration.
3. Create cluster.
4. Roll out Deployment.
5. Test Service.
6. Delete cluster.

Acceptance:

kind get clusters
kubectl get nodes -o wide
kubectl get pods -A

Lab 2: minikube with Ingress

Goal: Start minikube, enable ingress, and reach an app through a hostname.

Tasks:

1. Start minikube.
2. Enable ingress add-on.
3. Deploy demo app.
4. Create Ingress.
5. Test host resolution.

Acceptance:

kubectl get ingress -A
curl -H 'Host: web.localtest.me' http://$(minikube ip)

Lab 3: kubeadm Single-Node in VirtualBox

Goal: Install Kubernetes manually in a VM.

Tasks:

1. Create Ubuntu VM.
2. Install containerd.
3. Install kubeadm/kubelet/kubectl.
4. Run kubeadm init.
5. Install CNI.
6. Remove control-plane taint.
7. Test demo app.

Acceptance:

kubectl get nodes -o wide
kubectl get pods -A
kubectl run test --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default

Lab 4: kubeadm Multi-Node with Vagrant

Goal: Create three VMs reproducibly and build a cluster.

Tasks:

1. Create Vagrantfile.
2. Run common provisioning.
3. Initialize control plane.
4. Join workers.
5. Install CNI.
6. Test scheduling.

Acceptance:

vagrant status
kubectl get nodes -o wide
kubectl create deployment web -n demo --image=nginx:stable --replicas=6
kubectl get pods -n demo -o wide

Lab 5: Secure a Remote VM

Goal: Operate a remote single-node cluster with a minimal firewall.

Tasks:

1. Create remote VM.
2. Set up SSH key access.
3. Open firewall only for admin IP.
4. Install Kubernetes.
5. Validate API access.
6. Optionally open ingress.
7. Tear down.

Acceptance:

ssh workshop@<remote-ip> hostname
kubectl get nodes -o wide
sudo ufw status verbose

Lab 6: Cluster Failure Modes

Goal: Identify typical failures and fix them methodically.

Failure modes:

• CNI missing,
• wrong Service selector,
• ImagePullBackOff,
• Node NotReady,
• Ingress without controller,
• kubeconfig points to wrong IP.

Acceptance:

For each failure, document:

• symptom,
• diagnostic command,
• cause,
• correction,
• validation.

37. Review Checklist

37.1 Before Provisioning

• Target variant selected.
• Host resources are sufficient.
• CIDRs do not collide.
• IP plan documented.
• SSH access defined.
• Kubernetes version pinned.
• CNI selected.
• Teardown is known.
• No production secrets are used.

37.2 After OS Baseline

• Hostname unique.
• IP address correct.
• DNS or /etc/hosts correct.
• Time synchronized.
• Swap disabled.
• Kernel modules loaded.
• sysctl configured.
• containerd running.
• kubelet/kubeadm/kubectl installed.

37.3 After Cluster Bootstrap

• kubectl get nodes shows expected nodes.
• Nodes are Ready.
• CNI pods run.
• CoreDNS runs.
• Demo pod starts.
• Service discovery works.
• Events show no persistent critical errors.
• kubeconfig points to the correct API endpoint.

37.4 After Add-ons

• Metrics Server works.
• StorageClass exists, if needed.
• Ingress Controller or Gateway Controller runs.
• LoadBalancer concept is tested.
• NetworkPolicy behavior is documented.
• Security baseline is applied.

37.5 Before Handover to Participants

• Reset procedure tested.
• Snapshots clearly named.
• Download links and images reachable.
• Provider costs considered.
• Troubleshooting commands ready.
• Solution steps documented.

38. Anti-Patterns

39. Command Collection

39.1 Cluster Context

kubectl config get-contexts
kubectl config current-context
kubectl cluster-info
kubectl version

39.2 Nodes

kubectl get nodes -o wide
kubectl describe node <node>
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>

39.3 kubeadm

kubeadm version
sudo kubeadm init --config kubeadm-config.yaml
kubeadm token create --print-join-command
sudo kubeadm reset -f
sudo kubeadm upgrade plan

39.4 containerd and CRI

sudo systemctl status containerd --no-pager
sudo journalctl -u containerd -n 100 --no-pager
sudo crictl info
sudo crictl ps -a
sudo crictl images
sudo crictl logs <container-id>

39.5 kubelet

sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
sudo journalctl -u kubelet -f

39.6 kind

kind version
kind create cluster --name workshop
kind create cluster --config kind-workshop.yaml
kind get clusters
kind get nodes --name workshop
kind load docker-image workshop/app:dev --name workshop
kind delete cluster --name workshop

39.7 minikube

minikube version
minikube start --driver=docker
minikube start --driver=virtualbox
minikube start --driver=vmware
minikube start --driver=ssh --ssh-ip-address=<REMOTE_VM_IP>
minikube status
minikube addons list
minikube addons enable ingress
minikube tunnel
minikube stop
minikube delete --all --purge

39.8 Vagrant

vagrant version
vagrant plugin list
vagrant up --provider=virtualbox
vagrant up --provider=vmware_desktop
vagrant status
vagrant ssh cp1
vagrant provision cp1
vagrant snapshot save cp1 before-change
vagrant destroy -f

39.9 Remote Diagnostics

ssh workshop@<ip> hostnamectl
ssh workshop@<ip> ip addr
ssh workshop@<ip> sudo ufw status verbose
nc -vz <api-endpoint> 6443
curl -k https://<api-endpoint>:6443/readyz

39.10 Network

ip addr
ip route
ss -tulpen
sudo tcpdump -ni any port 6443
kubectl get svc,endpoints,endpointslices -A
kubectl get ingress -A
kubectl get networkpolicy -A

40. Sources and Further Documentation

Primary sources you should check regularly for versions, flags, and provider details:

• Kubernetes Setup: https://kubernetes.io/docs/setup/
• kubeadm: https://kubernetes.io/docs/reference/setup-tools/kubeadm/
• Installing kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
• Creating a kubeadm cluster: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• kubeadm HA: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
• Kubernetes Ports and Protocols: https://kubernetes.io/docs/reference/networking/ports-and-protocols/
• Container Runtimes: https://kubernetes.io/docs/setup/production-environment/container-runtimes/
• kind Quick Start: https://kind.sigs.k8s.io/docs/user/quick-start/
• kind Configuration: https://kind.sigs.k8s.io/docs/user/configuration/
• kind Ingress: https://kind.sigs.k8s.io/docs/user/ingress/
• kind LoadBalancer: https://kind.sigs.k8s.io/docs/user/loadbalancer/
• kind Local Registry: https://kind.sigs.k8s.io/docs/user/local-registry/
• minikube Start: https://minikube.sigs.k8s.io/docs/start/
• minikube Drivers: https://minikube.sigs.k8s.io/docs/drivers/
• minikube VirtualBox: https://minikube.sigs.k8s.io/docs/drivers/virtualbox/
• minikube VMware: https://minikube.sigs.k8s.io/docs/drivers/vmware/
• minikube SSH: https://minikube.sigs.k8s.io/docs/drivers/ssh/
• Vagrant Documentation: https://developer.hashicorp.com/vagrant/docs
• Vagrant VirtualBox Provider: https://developer.hashicorp.com/vagrant/docs/providers/virtualbox
• Vagrant VMware Provider: https://developer.hashicorp.com/vagrant/docs/providers/vmware
• k3s Installation: https://docs.k3s.io/installation
• RKE2 Installation: https://docs.rke2.io/install/quickstart
• RKE2 HA: https://docs.rke2.io/install/ha
• k0s Installation: https://docs.k0sproject.io/
• Talos Linux Documentation: https://docs.siderolabs.com/talos/
• MetalLB: https://metallb.io/