1. Purpose of the Curriculum

This curriculum helps you understand Kubernetes not merely as a collection of individual commands, but as a platform model for operating containerized applications. You learn how applications are declared, rolled out, observed, secured, scaled, and analyzed when failures occur.

After completing the full program, you will be able to:

• Explain Kubernetes architecture and responsibilities.
• Read, write, validate, and operate Kubernetes resources.
• Roll out applications in a production-oriented way using Deployments, Services, ConfigMaps, Secrets, probes, and resource specifications.
• Analyze common failure modes using kubectl, logs, events, metrics, and resource status.
• Classify platform topics such as networking, storage, scheduling, add-ons, observability, GitOps, and backup.
• Apply security controls such as RBAC, ServiceAccounts, Pod Security Standards, NetworkPolicies, and secret protection.
• Document architecture decisions and make Kubernetes solutions reviewable.

1.1. What This Curriculum Is Intended to Provide

This curriculum is designed for workshops, internal training, bootcamps, and structured self-learning phases. It combines conceptual explanations with hands-on exercises. Every learning unit should relate to real operations: What happens in the cluster? How do you verify state? Which risks arise? How can you roll back changes?

1.2. What This Curriculum Deliberately Does Not Replace

This curriculum does not replace your organization’s production platform documentation, provider-specific operations manuals, or a complete security policy. It provides the professional structure you can use to create, review, or improve those documents.

2. Didactic Model

The workshop follows a recurring learning cycle:

Understand -> Apply -> Observe -> Debug -> Secure -> Document

Whenever possible, each topic is handled in five steps:

2.1. Learning Principles

• Think declaratively: You describe desired states instead of manually wiring processes together inside the cluster.
• Read status: You learn to use spec, status, conditions, and events as diagnostic sources.
• Create failures deliberately: You learn faster when you reproduce common failure modes in a controlled way.
• Include operations: Every exercise includes validation, teardown, and a brief operational assessment.
• Do not bolt security on later: Security aspects are carried throughout from the beginning, not added only at the end.
• Make work reviewable: Every solution should be understandable, versionable, and verifiable.

2.2. Depth per Topic

Not every topic needs the same depth. For beginners, mental models and safe standard paths are sufficient. Platform teams additionally need upgrade, failure, security, and operational perspectives.

3. Target Audiences and Roles

The curriculum is modular. You can tailor it to different roles.

3.1. Expected Outcomes by Role

Developers

After the training, you should be able to roll out an application cleanly on Kubernetes. This includes a Deployment, Service, configuration, probes, resource specifications, logs, and a simple rollback.

DevOps and System Engineers

You should be able to operate workloads in a production-oriented way. This includes debugging, rollout strategies, storage, Ingress, DNS, resource management, Helm or Kustomize, and baseline security.

Platform Engineers and SREs

You should be able to assess platform decisions. This includes cluster architecture, add-ons, GitOps, observability, multi-tenancy, upgrade strategies, backup, and disaster recovery.

Security Engineers

You should be able to identify Kubernetes-specific attack surfaces and implement technical controls. This includes RBAC, ServiceAccounts, admission, Pod Security Standards, NetworkPolicies, secret protection, image security, and runtime detection.

4. Prerequisites

4.1. Conceptual Prerequisites

For entry-level participation, you need:

• Basic understanding of containers and images.
• Basic Linux shell skills.
• Basic understanding of HTTP, DNS, ports, and TCP/IP.
• Basic YAML knowledge.
• Ability to read error messages systematically.

For Professional and Advanced topics, you should additionally bring:

• Experience with CI/CD or GitOps.
• Experience with Linux system services and logs.
• Understanding of TLS, certificates, authentication, and authorization.
• Basic understanding of cloud networking, load balancing, and storage.
• Experience with monitoring and logging systems.

4.2. Technical Prerequisites

For the labs, you need at least:

• kubectl
• access to a Kubernetes cluster
• a shell environment
• Git
• an editor
• optional: Helm, Kustomize, jq, yq, stern, k9s, crictl

Depending on the workshop goal, the following are suitable for local workshops:

• kind
• minikube
• k3d
• k3s
• kubeadm-based VMs
• managed Kubernetes clusters from cloud providers

4.3. Preparation Check

Before the workshop, you should verify:

kubectl version --client
kubectl config current-context
kubectl cluster-info
kubectl get nodes -o wide
kubectl auth can-i get pods -A

For most exercises, you need a namespace in which you are allowed to create, modify, and delete resources.

5. Learning Paths

5.1. Learning Path for Developers

Goal: You can deploy, configure, verify, and analyze your application on Kubernetes.

Recommended sequence:

1. Kubernetes core model
2. Pods, Deployments, and ReplicaSets
3. Services and DNS
4. ConfigMaps and Secrets
5. Probes and rollouts
6. Logs, events, and debugging
7. Resource specifications and basic scaling
8. Final exercise: production-oriented application rollout

5.2. Learning Path for DevOps and System Engineers

Goal: You can operate workloads, analyze failures, and establish robust standards for teams.

Recommended sequence:

1. Complete Fundamentals
2. Workload controllers in detail
3. Resource management and scheduling
4. Ingress, Gateway API, and NetworkPolicies
5. Storage and stateful workloads
6. Helm, Kustomize, and CI/CD validation
7. Observability and incident triage
8. Final exercise: stabilize a faulty application

5.3. Learning Path for Platform Engineers

Goal: You can plan, operate, and evolve Kubernetes platforms.

Recommended sequence:

1. Cluster architecture and Control Plane
2. kubeadm, managed Kubernetes, and distributions
3. CNI, CSI, Ingress, and load balancing
4. Observability stack
5. GitOps and platform add-ons
6. Multi-tenancy, quotas, policies, and cost control
7. Backup, restore, and disaster recovery
8. Final exercise: create a platform blueprint

5.4. Learning Path for Security Engineers

Goal: You can harden Kubernetes workloads and platforms, detect attacks, and handle incidents in a structured way.

Recommended sequence:

1. Kubernetes threat model
2. API access, authentication, and authorization
3. RBAC least privilege
4. ServiceAccounts and token behavior
5. Pod Security Standards and Admission Control
6. SecurityContext, capabilities, seccomp, AppArmor, and SELinux
7. NetworkPolicies and service-mesh/CNI policies
8. Secret protection, image security, and runtime detection
9. Final exercise: security review and incident simulation

6. Competency Levels

6.1. Level A: Orientation

You can describe what Kubernetes does, what a cluster is used for, and which core components are involved.

Expected competency:

• You can explain the Control Plane and Worker Nodes.
• You can distinguish Pod, Deployment, and Service.
• You understand Desired State and Reconciliation.
• You can display a namespace and simple resources.

6.2. Level B: Application

You can create, roll out, and verify simple workloads.

Expected competency:

• You can write simple manifests.
• You can safely use kubectl apply, get, describe, logs, and delete.
• You expose an application through a Service.
• You inspect events, logs, and resource state.

6.3. Level C: Production-Oriented Workloads

You can configure workloads robustly.

Expected competency:

• You define requests, limits, and probes.
• You use ConfigMaps and Secrets deliberately.
• You control rollouts and rollbacks.
• You understand HPA, PDB, and simple Ingress scenarios.
• You recognize common failure modes.

6.4. Level D: Platform Operations

You can assess cluster and platform decisions.

Expected competency:

• You understand Control Plane HA, etcd, CNI, CSI, and add-ons.
• You evaluate distributions and managed Kubernetes.
• You plan observability, GitOps, backup, restore, and upgrade processes.
• You define multi-tenancy standards.

6.5. Level E: Security and Resilience

You can deliberately harden Kubernetes and handle incident paths.

Expected competency:

• You implement RBAC least privilege.
• You inspect ServiceAccounts and tokens.
• You use Pod Security Standards and SecurityContexts.
• You assess NetworkPolicies, image security, secret protection, and runtime detection.
• You perform structured incident analysis.

7. Overall Workshop Structure

The curriculum is divided into four main levels:

7.1. Recommended Module Sequence

flowchart TD
  A[Fundamentals] --> B[Intermediate]
  B --> C[Professional]
  B --> D[Advanced and Security]
  C --> E[Platform Project]
  D --> F[Security Project]
  E --> G[Final Assessment]
  F --> G

7.2. Minimum Path for a Compact Workshop

When time is limited, the workshop should include at least these topics:

1. Cluster architecture and resource model
2. Namespace, Pod, Deployment, ReplicaSet, and Service
3. ConfigMap, Secret, probes, and resource specifications
4. Logs, events, describe, and rollback
5. RBAC fundamentals and Pod Security fundamentals
6. Final exercise with Deployment, Service, and failure analysis

8. Level 1: Fundamentals

8.1. Goal

You understand the basic idea of Kubernetes and can deploy, expose, and verify simple applications.

8.2. Target Audience

• Beginners
• Developers
• Junior DevOps
• Administrators with limited Kubernetes experience
• Teams that will use Kubernetes as a target platform

8.3. Learning Objectives

After this level, you can:

• Explain Kubernetes as an orchestration platform.
• Distinguish Control Plane and Worker Nodes.
• Describe the role of the API Server, etcd, Scheduler, Controller Manager, kubelet, and kube-proxy.
• Explain Desired State, Actual State, and Reconciliation.
• Classify Namespaces, Pods, Deployments, ReplicaSets, and Services.
• Apply labels and selectors.
• Use kubectl to display, create, and delete resources.
• Use logs and events for basic failure analysis.

8.4. Module F1: Introduction and Mental Model

Content:

• Container orchestration
• desired state and actual state
• declarative configuration
• Kubernetes API as the central interface
• controllers as continuous control loops
• boundaries relative to Docker, CI/CD, monitoring, and cloud providers

Exercise:

kubectl version --client
kubectl cluster-info
kubectl config current-context
kubectl get nodes -o wide
kubectl create namespace workshop
kubectl get namespace workshop

Reflection questions:

• Why is a Kubernetes manifest not a script?
• What happens when the desired state does not match the actual state?
• Why is the API the central integration point?

8.5. Module F2: Cluster Architecture

Content:

• Control Plane
• Worker Nodes
• API Server
• etcd
• Scheduler
• Controller Manager
• Cloud Controller Manager
• kubelet
• Container Runtime
• kube-proxy or eBPF dataplane
• CNI plugin
• CoreDNS

Exercise:

kubectl get nodes -o wide
kubectl get namespaces
kubectl get pods -n kube-system -o wide
kubectl describe node <node-name>

Common pitfalls:

• You confuse Control Plane components with workload components.
• You expect a Pod to be started directly by kubectl on a Node.
• You overlook that controllers continuously reconcile state.

8.6. Module F3: Resource Model and YAML

Content:

• API groups and kinds
• object metadata
• Namespaces
• labels and annotations
• spec as desired state
• status as system feedback
• conditions and events
• resource identity through name, namespace, and UID

Example:

apiVersion: v1
kind: Namespace
metadata:
  name: workshop
  labels:
    purpose: training

Exercise:

kubectl apply -f namespace.yaml
kubectl get namespace workshop -o yaml
kubectl explain pod.spec.containers

8.7. Module F4: Pods

Content:

• Pod as a group of one or more containers
• shared network identity
• container ports
• volumes in the Pod
• restart policy
• Pod phases
• container states
• events as a failure source

Exercise:

kubectl run nginx-demo -n workshop --image=nginx:1.27 --restart=Never
kubectl get pod nginx-demo -n workshop -o wide
kubectl describe pod nginx-demo -n workshop
kubectl logs nginx-demo -n workshop
kubectl delete pod nginx-demo -n workshop

Key insight:

A Pod is usually not a long-term manually managed object. For normal applications, you use controllers such as Deployments.

8.8. Module F5: Deployments and ReplicaSets

Content:

• Deployment as a controller for stateless applications
• ReplicaSet as a replication mechanism
• Pod template
• selectors
• rolling update
• revisions
• rollback

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: workshop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80

Exercise:

kubectl apply -f deployment.yaml
kubectl get deployment,replicaset,pod -n workshop
kubectl rollout status deployment/web -n workshop
kubectl scale deployment/web -n workshop --replicas=3
kubectl rollout history deployment/web -n workshop

8.9. Module F6: Services and Service Discovery

Content:

• Pod IP as an ephemeral address
• Service as a stable abstraction
• label selector
• ClusterIP
• NodePort
• LoadBalancer
• Endpoints and EndpointSlices
• CoreDNS
• Service DNS names

Example:

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: workshop
spec:
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: 80
  type: ClusterIP

Exercise:

kubectl apply -f service.yaml
kubectl get service,endpoints,endpointslice -n workshop
kubectl run dns-test -n workshop --rm -it --image=busybox:1.36 --restart=Never -- sh
wget -qO- http://web.workshop.svc.cluster.local

8.10. Module F7: kubectl Fundamentals

Content:

• context and kubeconfig
• Namespaces
• get
• describe
• logs
• apply
• delete
• diff
• explain
• output formats
• field selector and label selector

Exercise:

kubectl get all -n workshop
kubectl get pods -n workshop -l app=web
kubectl describe deployment web -n workshop
kubectl logs deployment/web -n workshop
kubectl diff -f deployment.yaml
kubectl delete -f deployment.yaml

8.11. Module F8: Fundamentals Final Exercise

Exercise goal:

You roll out a simple web application, publish it internally in the cluster, inspect logs and events, and document teardown.

Tasks:

1. Create a namespace.
2. Create a Deployment with two replicas.
3. Create a ClusterIP Service.
4. Inspect Pods, ReplicaSets, Deployment, and Service.
5. Test DNS from a debug Pod.
6. Change the image and observe the rollout.
7. Perform a rollback.
8. Delete all resources cleanly.

Acceptance criteria:

• The namespace is clearly named.
• The Deployment has labels and matching selectors.
• The Service points to the expected Pods.
• The rollout is traceable.
• The teardown is documented.

9. Level 2: Intermediate

9.1. Goal

You can configure, debug, and operate production-oriented workloads. The focus is robustness, configuration, resources, rollouts, networking, storage, and operational failure analysis.

9.2. Target Audience

• DevOps engineers
• Developers with operational responsibility
• System engineers
• Kubernetes users with foundational knowledge
• Teams that want to operate applications on Kubernetes in a production-oriented way

9.3. Learning Objectives

After this level, you can:

• Distinguish Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs.
• Use ConfigMaps and Secrets correctly.
• Configure liveness, readiness, and startup probes.
• Explain requests, limits, and QoS classes.
• Classify HPA and PDB.
• Apply Ingress, Gateway API, and DNS fundamentals.
• Use PV, PVC, and StorageClass.
• Understand scheduling rules such as Node Affinity, Pod Affinity, taints, and tolerations.
• Use Helm and Kustomize for packaging and variant management.
• Analyze common failure modes systematically.

9.4. Module I1: Workload Controllers Compared

Exercise:

• Create a Deployment for a web application.
• Create a Job for a one-time task.
• Create a CronJob with a short test frequency.
• Inspect generated Pods and logs.
• Delete the test resources.

Acceptance criteria:

• You can explain why a Deployment is not suitable for every use case.
• You recognize when stable identity or scheduled execution matters more than horizontal scaling.

9.5. Module I2: Configuration with ConfigMaps and Secrets

Content:

• separate configuration from images
• ConfigMaps as files or environment variables
• Secrets as Kubernetes resources
• Base64 is not encryption
• rollout behavior when configuration changes
• restrict Secret access with RBAC
• classify external secret systems

Exercise:

kubectl create configmap app-config \
  -n workshop \
  --from-literal=LOG_LEVEL=info \
  --dry-run=client -o yaml > configmap.yaml

kubectl create secret generic app-secret \
  -n workshop \
  --from-literal=API_TOKEN=dummy \
  --dry-run=client -o yaml > secret.yaml

Tasks:

1. Mount a ConfigMap as an environment variable.
2. Mount a Secret as a volume.
3. Verify whether the application sees the values.
4. Change the ConfigMap.
5. Observe whether a rollout happens automatically.
6. Document how to trigger a deliberate rollout.

9.6. Module I3: Probes and Application Lifecycle

Content:

• difference between “process is running” and “application is ready”
• HTTP, TCP, and exec probes
• initial delay, timeout, period, and failure threshold
• graceful shutdown
• terminationGracePeriodSeconds
• preStop hook
• impact on rollouts

Exercise:

1. Add a readiness probe.
2. Add a liveness probe.
3. Create a faulty probe.
4. Observe events and container restarts.
5. Correct the probe.

Failure mode:

An overly aggressive liveness probe can keep restarting an application that is still starting up. Use startup probes or realistic thresholds.

9.7. Module I4: Resource Management

Content:

• CPU requests
• memory requests
• CPU limits
• memory limits
• QoS classes: Guaranteed, Burstable, BestEffort
• OOMKilled
• NodePressure
• LimitRange
• ResourceQuota
• Metrics Server
• Horizontal Pod Autoscaler

Example:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"

Exercise:

1. Add requests and limits to a Deployment.
2. Inspect the resulting QoS class.
3. Set a ResourceQuota in the namespace.
4. Deliberately create a manifest that violates the quota.
5. Analyze the error message.

Acceptance criteria:

• You can explain why missing requests degrade scheduling and capacity planning.
• You can distinguish OOMKilled from CrashLoopBackOff.
• You understand that limits do not automatically mean good performance.

9.8. Module I5: Rollouts, Rollbacks, and Availability

Content:

• Deployment strategy
• RollingUpdate
• maxUnavailable
• maxSurge
• revision history
• rollout status
• rollback
• PodDisruptionBudget
• readiness as a rollout signal

Exercise:

kubectl rollout status deployment/web -n workshop
kubectl rollout history deployment/web -n workshop
kubectl set image deployment/web nginx=nginx:1.28 -n workshop
kubectl rollout undo deployment/web -n workshop

Tasks:

1. Change an image to a valid version.
2. Observe the rolling update.
3. Change an image to an invalid version.
4. Analyze the stuck rollout.
5. Perform a rollback.
6. Add a PodDisruptionBudget.

9.9. Module I6: Networking, DNS, Ingress, and Gateway API

Content:

• Pod network
• Service Discovery
• ClusterIP, NodePort, LoadBalancer
• DNS names for Services
• Endpoints and EndpointSlices
• Ingress Controller
• Ingress resources
• Gateway API as an extensible routing model
• TLS termination
• NetworkPolicies

Exercise:

1. Inspect Service and EndpointSlices.
2. Test DNS from a debug Pod.
3. Create a simple Ingress rule or HTTPRoute if the environment supports it.
4. Add a deny-by-default NetworkPolicy.
5. Explicitly allow traffic from a specific Pod label.

Acceptance criteria:

• You can explain why a Service without Endpoints cannot work.
• You can distinguish DNS problems from Service selector problems.
• You know that NetworkPolicies are only enforced if the network plugin supports them.

9.10. Module I7: Storage

Content:

• temporary Pod storage
• Volumes
• PersistentVolume
• PersistentVolumeClaim
• StorageClass
• dynamic provisioning
• access modes
• reclaim policy
• CSI
• volume expansion
• backup and restore

Exercise:

1. Create a PVC.
2. Mount it into a Pod or Deployment.
3. Write test data.
4. Delete the Pod.
5. Verify whether the data remains.
6. Inspect PV, PVC, StorageClass, and events.

Acceptance criteria:

• You distinguish Pod lifetime from data lifetime.
• You can analyze PVC Pending.
• You can explain why stateful workloads require backup and restore design.

9.11. Module I8: Scheduling

Content:

• Node Selector
• Node Affinity
• Pod Affinity
• Pod Anti-Affinity
• taints
• tolerations
• Topology Spread Constraints
• PriorityClass
• preemption
• resource requests as the foundation for scheduling

Exercise:

1. Assign a label to a Node, if permitted.
2. Schedule a Pod onto matching Nodes using Node Selector.
3. Use a toleration for a tainted Node.
4. Deliberately create a Pending Pod.
5. Analyze Scheduler events.

Common misconception:

Scheduling rules are not security boundaries. They control placement, but they do not replace RBAC, admission, Pod Security, or network isolation.

9.12. Module I9: Helm and Kustomize

Content:

• purpose of packaging and templating
• Helm chart structure
• values files
• Helm template and dry run
• Helm release lifecycle
• Kustomize bases and overlays
• patches
• generators
• validation in CI/CD

Exercise:

helm lint ./chart
helm template demo ./chart -f values.yaml
helm upgrade --install demo ./chart -n workshop --dry-run
kubectl kustomize overlays/dev
kubectl apply -k overlays/dev --dry-run=server

Acceptance criteria:

• You can explain when Helm is useful.
• You can explain when Kustomize is simpler and more transparent.
• You validate generated manifests before applying them.

9.13. Module I10: Debugging and Triage

Content:

• kubectl get
• kubectl describe
• logs from current and previous containers
• events
• debug Pods
• ephemeral containers
• port forwarding
• Service and DNS tests
• resource status
• CrashLoopBackOff
• ImagePullBackOff
• Pending
• OOMKilled
• Forbidden

Exercise:

You receive a deliberately broken application. You must document the cause, correction, and validation.

Possible failures:

• wrong image
• missing Secret key
• wrong Service selector
• memory limits too low
• faulty readiness probe
• missing RBAC permission
• PVC without matching StorageClass

Acceptance criteria:

• You document symptoms and causes separately.
• You support the cause with logs, events, or status fields.
• You perform a minimal correction.
• You verify success after the correction.

10. Level 3: Professional

10.1. Goal

You can plan, operate, and assess Kubernetes platforms. The focus is cluster architecture, distributions, lifecycle, networking, storage, observability, GitOps, multi-tenancy, cost, and disaster recovery.

10.2. Target Audience

• Platform engineers
• Cloud engineers
• SREs
• system architects
• tech leads
• experienced DevOps engineers

10.3. Learning Objectives

After this level, you can:

• Assess cluster architectures.
• Classify distributions and operating models.
• Explain Control Plane HA and etcd risks.
• Account for Kubernetes versions and version skew.
• Evaluate CNI, CSI, and load balancing.
• Plan observability stacks for Kubernetes.
• Classify GitOps with Argo CD or Flux.
• Operate platform add-ons in a controlled way.
• Design multi-tenancy using Namespaces, RBAC, quotas, and NetworkPolicies.
• Plan backup, restore, and disaster recovery.

10.4. Module P1: Cluster Architecture and Operating Model

Content:

• single-cluster vs multi-cluster
• multi-zone and multi-region
• Control Plane HA
• worker node pools
• separated system and workload node pools
• cluster per environment
• cluster per tenant
• shared cluster with Namespaces
• Cluster API and lifecycle automation
• managed Kubernetes vs self-operated clusters

Decision questions:

• Who operates the Control Plane?
• Who patches Nodes?
• Who owns CNI, CSI, and Ingress?
• How is cluster state backed up?
• How are platform changes tested?
• What isolation does your tenant model require?

Exercise:

Create a target cluster architecture for an example scenario. Document:

• operating model
• environments
• node pools
• network assumptions
• storage requirements
• upgrade path
• backup and restore concept
• risks and open decisions

10.5. Module P2: Distributions and Installation Paths

Content:

• kubeadm
• managed Kubernetes
• RKE2
• k3s
• k0s
• OpenShift
• Talos
• kind and minikube as learning and test environments
• productivity and operational differences
• support model
• upgrade path
• add-on responsibility

Assessment criteria:

10.6. Module P3: kubeadm and Cluster Lifecycle

Content:

• purpose of kubeadm
• kubeadm init
• kubeadm join
• certificates
• bootstrap tokens
• kubeconfig files
• Control Plane expansion
• worker join
• upgrade planning
• version skew
• reset and recovery
• limits of kubeadm

Exercise:

In a suitable lab environment:

1. Verify prerequisites for Nodes.
2. Initialize a Control Plane.
3. Install a CNI plugin.
4. Add a Worker Node.
5. Verify cluster state.
6. Simulate a certificate or join problem.
7. Document the correction.

Note:

kubeadm is a bootstrapping tool. Machine provisioning, fully automated add-on management, and complete platform operations must be handled through additional tools or processes.

10.7. Module P4: etcd and Control Plane Operations

Content:

• etcd as a consistent key-value store
• snapshot and restore
• quorum
• latency and storage performance
• certificates
• defragmentation
• monitoring
• backup frequency
• restore test
• Control Plane components as static Pods

Exercise:

1. Inspect Control Plane Pods.
2. Create an etcd snapshot in a lab environment.
3. Document restore steps.
4. Define a restore test interval.
5. Formulate risks of a faulty etcd backup.

Acceptance criteria:

• You can explain why a backup without a restore test is insufficient.
• You know the impact of quorum loss.
• You distinguish application backup from cluster state backup.

10.8. Module P5: CNI, Service Network, and Load Balancing

Content:

• CNI responsibilities
• Pod network
• Service network
• kube-proxy
• eBPF datapanes
• ClusterIP implementation
• LoadBalancer integration
• Ingress Controller
• Gateway Controller
• NetworkPolicy support
• DNS and CoreDNS
• Service Mesh as an optional additional model

Exercise:

Analyze the network path of a request:

Client -> Load Balancer -> Ingress/Gateway -> Service -> EndpointSlice -> Pod

For each hop, document:

• involved Kubernetes resource
• involved platform component
• possible failure source
• suitable diagnostic commands

10.9. Module P6: CSI and Storage Operations

Content:

• CSI drivers
• StorageClasses
• dynamic provisioning
• reclaim policies
• volume binding mode
• snapshots
• volume expansion
• performance classes
• backup and restore for stateful workloads
• data residency
• encryption

Exercise:

Create a storage design for three workload types:

1. stateless API
2. database
3. message broker

Assess each for:

• persistence need
• backup objective
• restore time
• performance need
• availability requirement
• risks during Node failure

10.10. Module P7: Observability

Content:

• metrics
• logs
• traces
• events
• Kubernetes object status
• Prometheus
• Grafana
• Loki
• Tempo
• OpenTelemetry
• alerting
• SLOs and SLIs
• golden signals
• runbooks

Exercise:

Design an observability model for an example application:

• Which metrics do you need?
• Which logs do you need?
• Which traces help with failure analysis?
• Which Kubernetes events are relevant?
• Which alerts are useful?
• Which alerts would only create noise?

Acceptance criteria:

• You separate infrastructure, platform, and application signals.
• You define alerts with an action path.
• You avoid pure symptom metrics without context.

10.11. Module P8: GitOps and Delivery

Content:

• Git as desired state
• pull-based deployment
• Argo CD
• Flux
• drift detection
• sync waves
• health checks
• secrets in the GitOps model
• environment variants
• release promotion
• rollback through Git
• policy gates in CI/CD

Exercise:

1. Structure a GitOps repository for Dev, Stage, and Prod.
2. Define which files are maintained by teams.
3. Define which files are platform standards.
4. Define validation steps.
5. Describe the rollback path.

10.12. Module P9: Platform Add-ons

Typical add-ons:

• Metrics Server
• Ingress Controller
• Gateway Controller
• cert-manager
• ExternalDNS
• Cluster Autoscaler
• Karpenter or alternative node autoscaling
• CNI plugin
• CSI driver
• observability components
• policy engines
• External Secrets Operator
• Argo CD or Flux

Assessment per add-on:

10.13. Module P10: Multi-Tenancy

Content:

• Namespace strategy
• RBAC per team
• ResourceQuota
• LimitRange
• NetworkPolicies
• Pod Security Admission
• separate node pools
• RuntimeClass
• Admission Policies
• Secret isolation
• logging and monitoring isolation
• cost centers and labels

Exercise:

Design a tenant model for three teams:

• Team A: internal API
• Team B: batch workloads
• Team C: sensitive data processing

Document:

• Namespaces
• roles and permissions
• quotas
• network rules
• Pod Security level
• exceptions and approval process

10.14. Module P11: Cost and Capacity

Content:

• requests as the basis for planning
• overcommitment
• idle capacity
• namespace cost
• labels for cost centers
• HPA, VPA, and Cluster Autoscaler
• spot or preemptible Nodes
• permissions and risks of autoscaling
• cost metrics

Exercise:

Analyze a namespace:

• sum of CPU requests
• sum of memory requests
• actual usage
• unused capacity
• workloads without requests
• possible optimizations

10.15. Module P12: Backup, Restore, and Disaster Recovery

Content:

• etcd backup
• application data backup
• Kubernetes manifests
• Secrets
• PV snapshots
• restore sequence
• RPO and RTO
• cluster rebuild
• GitOps rehydration
• disaster recovery tests

Exercise:

Create a DR concept for an application with a database:

• What is backed up?
• How often is it backed up?
• Where is it backed up?
• Who is allowed to restore?
• How is restore tested?
• What sequence applies during recovery?

11. Level 4: Advanced and Security

11.1. Goal

You can harden Kubernetes, assess risks, detect attacks, and handle incidents in a structured way. The focus is API access, RBAC, admission, Pod Security, network segmentation, secret protection, image supply chain, runtime detection, and forensics.

11.2. Target Audience

• Security engineers
• DevSecOps engineers
• Platform engineers
• SREs
• incident response teams
• architects with security responsibility

11.3. Learning Objectives

After this level, you can:

• Create a Kubernetes threat model.
• Distinguish authentication, authorization, and admission.
• Implement and verify RBAC least privilege.
• Design ServiceAccounts and tokens securely.
• Apply Pod Security Standards.
• Assess SecurityContexts.
• Classify capabilities, seccomp, AppArmor, and SELinux.
• Use NetworkPolicies and CNI-specific policies.
• Manage Secrets more securely.
• Reduce image and supply-chain risks.
• Structure runtime security and incident response.

11.4. Module S1: Kubernetes Threat Modeling

Content:

• assets in the cluster
• attack surfaces
• API Server as a central target
• ServiceAccount tokens
• Secrets
• container escape risks
• privileged Pods
• lateral movement
• supply-chain risks
• unsafe admission exceptions
• uncontrolled cluster-admin growth

Exercise:

Create a threat model for an example platform. Document:

• assets requiring protection
• possible attackers
• entry points
• escalation paths
• technical controls
• detection signals
• open risks

11.5. Module S2: Authentication, Authorization, and Admission

Content:

• Authentication: Who are you?
• Authorization: What are you allowed to do?
• Admission: Is this change allowed or will it be mutated?
• users, groups, and ServiceAccounts
• RBAC
• webhook authorization
• mutating admission webhooks
• validating admission webhooks
• ValidatingAdmissionPolicy
• audit logs

Exercise:

1. Check your permissions with kubectl auth can-i.
2. Create a role with minimal permissions.
3. Bind it to a ServiceAccount.
4. Test allowed and forbidden actions.
5. Document expected Forbidden errors.

11.6. Module S3: RBAC Least Privilege

Content:

• Role
• ClusterRole
• RoleBinding
• ClusterRoleBinding
• verbs
• resources
• API groups
• namespaced vs cluster-wide
• aggregated ClusterRoles
• impersonation
• dangerous permissions such as *, create pods, bind, escalate, impersonate

Exercise:

Create an RBAC model for an application:

• Deployment may read ConfigMaps.
• Deployment may not list Secrets.
• Operator may manage only resources in its own namespace.
• Developers may read logs but not modify ClusterRoles.

Acceptance criteria:

• You avoid broad ClusterRoleBindings.
• You can verify permissions with kubectl auth can-i --as.
• You document why a permission is required.

11.7. Module S4: ServiceAccounts and Tokens

Content:

• default ServiceAccount
• automatic token projection
• bound ServiceAccount tokens
• token lifetime
• audience
• workload identity models
• access from Pods
• minimizing API access

Exercise:

1. Create a dedicated ServiceAccount.
2. Assign it to a Deployment.
3. Remove unnecessary API permissions.
4. Inspect token mounting.
5. Disable automatic token mounting when it is not needed.

11.8. Module S5: Pod Security Standards and Admission

Pod Security Standards define three policy levels:

Content:

• Pod Security Admission
• Namespace labels for enforce, audit, and warn
• Baseline vs Restricted
• exceptions
• migration of old workloads
• distinction from the former PodSecurityPolicy
• combination with policy engines

Exercise:

1. Enable Pod Security Admission through Namespace labels in a lab environment.
2. Create a Pod that violates Restricted.
3. Analyze the warning or rejection.
4. Harden the Pod step by step.

11.9. Module S6: SecurityContext and Container Hardening

Content:

• runAsNonRoot
• runAsUser
• runAsGroup
• fsGroup
• readOnlyRootFilesystem
• allowPrivilegeEscalation
• Linux capabilities
• seccomp
• AppArmor
• SELinux
• privileged containers
• hostNetwork, hostPID, and hostIPC
• hostPath risks

Example:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop:
      - ALL

Exercise:

1. Inspect an insecure Pod.
2. Add SecurityContext settings.
3. Remove unnecessary capabilities.
4. Verify compatibility with the application.
5. Document necessary exceptions.

11.10. Module S7: Network Isolation

Content:

• default allow vs deny-by-default
• Ingress and Egress rules
• Namespace selector
• Pod selector
• IPBlock
• DNS exceptions
• CNI-specific extensions
• Cilium policies
• Layer 7 policies
• Service Mesh and mTLS

Exercise:

1. Create a deny-by-default policy in the namespace.
2. Allow traffic from frontend to backend.
3. Allow DNS traffic to CoreDNS.
4. Test allowed and blocked connections.
5. Document expected behavior.

Acceptance criteria:

• You verify whether your CNI enforces NetworkPolicies.
• You test rules in practice.
• You account for DNS and external dependencies.

11.11. Module S8: Secrets Security

Content:

• Kubernetes Secrets
• Base64 and encryption
• encryption at rest
• RBAC on Secrets
• Secret distribution
• external secret systems
• External Secrets Operator
• SOPS
• Vault
• Cloud Secret Manager
• rotation
• auditing
• accidental logging

Exercise:

Develop a Secret model:

• Which Secrets are stored in Kubernetes?
• Which Secrets remain external?
• Who may read Secrets?
• How are Secrets rotated?
• How do you prevent Secrets from ending up in Git or logs?

11.12. Module S9: Image Security and Supply Chain

Content:

• image tags vs digests
• minimal base images
• vulnerability scanning
• SBOM
• signatures
• Cosign
• basic SLSA concept
• admission policies for images
• private registries
• ImagePullSecrets
• provenance

Exercise:

1. Inspect an image’s tagging strategy.
2. Replace an imprecise tag with a digest.
3. Define a policy for allowed registries.
4. Describe how signature verification can be integrated into admission.

11.13. Module S10: Runtime Security

Content:

• suspicious process starts
• shell inside containers
• write access to system paths
• network connections
• access to ServiceAccount tokens
• Falco
• Tetragon
• eBPF-based signals
• RuntimeClass
• sandboxed runtimes

Exercise:

1. Define relevant runtime signals.
2. Map signals to real attack paths.
3. Design alerts with an action path.
4. Distinguish false positives from real incidents.

11.14. Module S11: Incident Response and Forensics

Content:

• incident classification
• containment
• evidence preservation
• preserving Pod and Node state
• logs, events, and audit logs
• suspicious RBAC changes
• suspicious images
• suspicious Secret access
• lateral movement
• recovery and lessons learned

Exercise:

You receive a compromised lab scenario. Work through:

1. initial triage
2. identify affected resources
3. check spread
4. plan containment
5. preserve evidence
6. derive root cause
7. define corrective measures
8. document follow-up

Acceptance criteria:

• You do not delete evidence prematurely.
• You separate containment from root cause analysis.
• You document decisions and assumptions.

12. Practice Environment

12.1. Variants

12.2. Minimum Requirements for the Lab Cluster

For Fundamentals:

• a functioning cluster
• one dedicated namespace per participant or team
• permissions for Pods, Deployments, Services, ConfigMaps, and Secrets
• access to logs and events

For Intermediate:

• Metrics Server
• Ingress Controller or Gateway Controller
• StorageClass
• ability to use NetworkPolicies
• optional Helm and Kustomize

For Professional:

• access to cluster add-ons
• observability system or demo stack
• ability to inspect Nodes or node pools
• optional kubeadm lab or managed Kubernetes sandbox

For Advanced and Security:

• RBAC test environment
• Pod Security Admission
• NetworkPolicy-capable CNI
• test registry or sample images
• optional Falco, Tetragon, Cilium, or policy engine

12.3. Naming Conventions

Recommended convention:

namespace: workshop-<name>
app:       <team>-<app>
labels:    app.kubernetes.io/name, app.kubernetes.io/part-of, app.kubernetes.io/component

Example:

metadata:
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: kubernetes-workshop
    app.kubernetes.io/component: frontend
    training.kubernetes.io/owner: team-a

12.4. Safety Rules for Labs

• Do not use real production Secrets.
• Use dedicated Namespaces.
• Delete lab resources after the exercise.
• Do not use cluster-admin privileges when namespace-level permissions are sufficient.
• Document intentional policy exceptions.
• Check the current context before destructive commands.

13. Workshop Labs

13.1. Lab 0: Setup and Orientation

Goal:

You ensure that your access works and that you are working in the correct cluster.

Tasks:

1. Check the local kubectl version.
2. Check the current context.
3. Display Nodes and Namespaces.
4. Create your workshop namespace.
5. Check your permissions.

Commands:

kubectl version --client
kubectl config current-context
kubectl get nodes
kubectl create namespace workshop-<name>
kubectl auth can-i create deployments -n workshop-<name>

Outcome:

You can state reliably which cluster and namespace you are working in.

13.2. Lab 1: Deploy the First Application

Goal:

You deploy a simple web application with Deployment and Service.

Tasks:

1. Create a Deployment.
2. Scale to two replicas.
3. Create a ClusterIP Service.
4. Inspect Pods, ReplicaSets, and Endpoints.
5. Test the application from a debug Pod.

Outcome:

The application is reachable internally inside the cluster.

13.3. Lab 2: Rollout and Rollback

Goal:

You control a change and roll back when failures occur.

Tasks:

1. Change the image to a new version.
2. Observe the rollout.
3. Set an invalid image.
4. Analyze the failure.
5. Perform a rollback.

Outcome:

You can detect and revert a faulty change.

13.4. Lab 3: Configuration and Secrets

Goal:

You separate application code from runtime configuration.

Tasks:

1. Create a ConfigMap.
2. Create a Secret with dummy values.
3. Mount both into a Deployment.
4. Verify values inside the Pod.
5. Change the configuration and trigger a rollout.

Outcome:

The application uses external configuration and no hard-coded values.

13.5. Lab 4: Probes and Resources

Goal:

You make an application more rollout-safe and easier to schedule.

Tasks:

1. Add a readiness probe.
2. Add a liveness probe.
3. Add a startup probe if startup time is relevant.
4. Set requests and limits.
5. Inspect events and QoS class.

Outcome:

The workload is easier to schedule and provides Kubernetes with meaningful health signals.

13.6. Lab 5: Ingress, Gateway, or External Access

Goal:

You make an application reachable over HTTP.

Tasks:

1. Check the available Ingress or Gateway controller.
2. Create a routing resource.
3. Configure hostname or path.
4. Check TLS assumptions.
5. Test access.

Outcome:

You understand the path from external traffic to the application.

13.7. Lab 6: Storage

Goal:

You use persistent data in Kubernetes.

Tasks:

1. Check StorageClasses.
2. Create a PVC.
3. Mount the PVC into a Pod.
4. Write data.
5. Delete and replace the Pod.
6. Check the data again.

Outcome:

You understand the difference between Pod lifetime and data lifetime.

13.8. Lab 7: RBAC

Goal:

You create minimal permissions for a ServiceAccount.

Tasks:

1. Create a ServiceAccount.
2. Create a Role with minimal permissions.
3. Create a RoleBinding.
4. Check permissions with kubectl auth can-i.
5. Test forbidden actions.

Outcome:

You can implement least privilege in practice.

13.9. Lab 8: NetworkPolicies

Goal:

You isolate workloads in the namespace.

Tasks:

1. Create a deny-by-default policy.
2. Explicitly allow frontend-to-backend traffic.
3. Allow DNS traffic.
4. Test allowed and blocked connections.
5. Document CNI assumptions.

Outcome:

You can control network communication in the cluster.

13.10. Lab 9: Failure Analysis

Goal:

You analyze several common failure modes.

Failure cases:

• Pod Pending
• CrashLoopBackOff
• ImagePullBackOff
• Service without Endpoints
• DNS resolution fails
• PVC Pending
• RBAC Forbidden
• OOMKilled
• stuck rollout

Procedure:

1. Describe the symptom.
2. Display relevant resources.
3. Check events.
4. Check logs.
5. Derive the cause.
6. Perform a minimal correction.
7. Validate success.

Outcome:

You can handle and document failures systematically.

13.11. Lab 10: Platform Blueprint

Goal:

You design a production-oriented Kubernetes platform.

Tasks:

1. Choose an operating model.
2. Define cluster and namespace structure.
3. Define add-ons.
4. Define observability.
5. Define security baseline.
6. Define backup and restore.
7. Define upgrade and rollback process.

Outcome:

You can justify architecture decisions.

13.12. Lab 11: Security Review

Goal:

You review an application for Kubernetes security risks.

Review points:

• ServiceAccount
• RBAC
• Secrets
• SecurityContext
• Pod Security level
• NetworkPolicies
• image source
• resource specifications
• host access
• privileges
• logs and sensitive data

Outcome:

You create a security finding with priority, risk, and correction proposal.

14. Final Projects

14.1. Project A: Production-Oriented App Rollout

Target audience:

• developers
• DevOps engineers
• Intermediate participants

Task:

You receive a simple application and must roll it out on Kubernetes in a production-oriented way.

Minimum requirements:

• Namespace
• Deployment with at least two replicas
• Service
• ConfigMap
• Secret with dummy values
• readiness and liveness probe
• requests and limits
• rollout and rollback documentation
• debugging evidence
• teardown instructions

Extensions:

• Ingress or Gateway API
• HPA
• PDB
• NetworkPolicy
• Helm or Kustomize

Assessment:

• functionality
• robustness
• traceability
• operational maturity
• security baseline

14.2. Project B: Platform Design

Target audience:

• Platform engineers
• SREs
• Tech leads
• Professional participants

Task:

You design a Kubernetes platform for multiple teams.

Minimum requirements:

• cluster strategy
• namespace and tenant model
• RBAC design
• quotas and LimitRanges
• network model
• Ingress or Gateway strategy
• storage strategy
• observability concept
• GitOps concept
• backup and DR concept
• upgrade strategy
• add-on responsibilities

Assessment:

• clear responsibility boundaries
• realistic operational assumptions
• upgradeability
• security level
• scalability
• clarity

14.3. Project C: Kubernetes Security Assessment

Target audience:

• Security engineers
• DevSecOps
• Advanced participants

Task:

You review an example cluster or namespace setup for security risks.

Minimum requirements:

• RBAC review
• ServiceAccount review
• Pod Security review
• SecurityContext review
• Secret review
• NetworkPolicy review
• image and registry review
• admission review
• runtime signal model
• short incident response plan

Assessment:

• risk assessment
• traceable evidence
• prioritization
• realistic correction proposals
• no unnecessarily destructive measures

15. Assessment Rubric

15.1. General Rubric

15.2. Passing Threshold

An exercise is considered passed when:

• functionality is demonstrably present,
• the most important resources are correctly named and explained,
• at least one validation step is documented,
• no real Secret was used,
• a teardown path exists.

15.3. Criteria for Excellent Solutions

A solution is excellent when it:

• is declarative and reproducible,
• uses clear labels and naming conventions,
• includes resource specifications,
• sets health checks meaningfully,
• accounts for rollout and rollback,
• sets SecurityContext and ServiceAccount deliberately,
• limits network access,
• documents relevant failure modes,
• was validated with kubectl diff or server-side dry run,
• does not require unnecessary cluster-level permissions.

16. Agenda Variants

16.1. Compact Workshop: 1 Day

Goal:

You get a practical introduction and can deploy simple workloads.

Not included:

• deep platform operations
• kubeadm lifecycle
• complete security hardening
• multi-tenancy design

16.2. Standard Workshop: 2 Days

Goal:

You can create production-oriented workloads and analyze common failures.

Day 1

Day 2

16.3. Professional Workshop: 3 Days

Goal:

You can operate workloads and make platform decisions.

Day 1

• Refresh Kubernetes fundamentals
• Deepen workload controllers
• Resources, probes, rollouts
• Debugging labs

Day 2

• Networking, Ingress, Gateway API
• Storage and CSI
• Scheduling
• Helm, Kustomize, and GitOps

Day 3

• Cluster architecture
• Distributions and kubeadm
• Observability
• Multi-tenancy
• Backup and DR
• Platform blueprint

16.4. Security Workshop: 2 Days

Goal:

You can identify and reduce Kubernetes-specific security risks.

Day 1

Day 2

16.5. Full Enablement Program: 6 to 8 Days

Goal:

You build sustainable Kubernetes competency in the team.

Recommended structure:

1. Day: Fundamentals
2. Day: Workloads and configuration
3. Day: Networking, storage, and scheduling
4. Day: Debugging, Helm, Kustomize, and delivery
5. Day: Platform architecture and kubeadm
6. Day: Observability, GitOps, multi-tenancy, and DR
7. Day: Security hardening
8. Day: Incident response, final project, and review

17. Trainer Guide

17.1. Preparation

Before the workshop, you should:

• test the cluster environment,
• prepare Namespaces or access,
• check permissions,
• validate example manifests,
• prepare intentional failure cases,
• document teardown commands,
• note provider-specific differences,
• test network and storage functions,
• plan Ingress or Gateway API only if the environment supports it.

17.2. Delivery

A good delivery follows this pattern:

1. Explain the concept briefly.
2. Show the resource in a manifest.
3. Apply the manifest.
4. Inspect status.
5. Provoke a failure.
6. Analyze the failure.
7. Perform the correction.
8. Document the result.

17.3. Handling Mixed Groups

In mixed groups, you should:

• support beginners with clear terminology,
• involve advanced participants through failure cases,
• mark platform topics as optional deep dives,
• capture security questions early but classify them in a structured way,
• split labs into base tasks and bonus tasks.

17.4. Common Time Traps

• local setup problems
• missing cluster permissions
• absent Ingress Controller
• missing StorageClass
• NetworkPolicies without a supporting CNI
• container images that cannot be pulled
• DNS or proxy problems in corporate networks
• overly large topic blocks without exercises

17.5. Recommended Scope Boundaries in the Workshop

Kubernetes should not be viewed in isolation from adjacent systems. Still, it must remain clear which tool carries which responsibility.

18. Knowledge Checks and Reflection Questions

18.1. Fundamentals

• What is the difference between a Pod and a Deployment?
• Why are Pods ephemeral?
• How does a Service find its Pods?
• What is the difference between spec and `status?
• Why is kubectl describe often more valuable than only kubectl get during failure analysis?
• How do you recognize whether a Service has Endpoints?
• Why should a manifest be versioned?

18.2. Intermediate

• When do you need a startup probe?
• How do readiness and liveness differ?
• What happens when a container exceeds its memory limit?
• Why are requests important for scheduling?
• Why does a ConfigMap change not always trigger a rollout automatically?
• How do you analyze a Pending Pod?
• How do you distinguish DNS, Service, and Endpoint problems?
• Which risks arise from missing NetworkPolicies?

18.3. Professional

• Which responsibilities remain yours when you use managed Kubernetes?
• Why is etcd especially critical?
• What does version skew mean?
• How do you plan a safe cluster upgrade?
• Which add-ons are critical for platform operations?
• How do you distinguish cluster backup from application data backup?
• When is multi-cluster useful?
• Which signals belong in a good Kubernetes observability model?

18.4. Advanced and Security

• Why is a ClusterRoleBinding with cluster-admin risky?
• Which permissions enable privilege escalation?
• What do Pod Security Standards provide, and what do they not provide?
• Why is Base64 not encryption?
• Which risks arise from hostPath?
• How do you test NetworkPolicies?
• Which indicators suggest a compromised Pod?
• What must you preserve before deleting a suspicious Pod?

19. Common Misconceptions

20. Recommended Duration

20.1. Duration by Depth

21. Mapping to Practical Roles

21.1. Developers

Required topics:

• Namespaces
• Pods
• Deployments
• Services
• ConfigMaps
• Secrets fundamentals
• probes
• resource specifications
• logs and events
• rollout and rollback

Optional topics:

• HPA
• Ingress or Gateway API
• Helm or Kustomize
• NetworkPolicies

21.2. DevOps Engineers

Required topics:

• all developer topics
• workload controllers
• storage
• scheduling
• Helm and Kustomize
• CI/CD validation
• debugging
• RBAC fundamentals
• PDB and HPA

Optional topics:

• GitOps
• observability
• kubeadm
• SecurityContext

21.3. Platform Engineers

Required topics:

• cluster architecture
• distributions
• CNI and CSI
• Ingress and Gateway API
• observability
• GitOps
• multi-tenancy
• add-ons
• backup and DR
• upgrade strategy
• cost control

Optional topics:

• runtime security
• Service Mesh
• Cluster API
• policy engines

21.4. Security Engineers

Required topics:

• threat modeling
• RBAC
• ServiceAccounts
• Pod Security Standards
• Admission Control
• SecurityContext
• NetworkPolicies
• Secrets security
• image security
• runtime detection
• incident response

Optional topics:

• Service Mesh security
• eBPF-based security
• node-level forensics
• compliance automation

22. Advanced Topics

These topics are suitable for deep-dive modules or separate workshops:

• Kubernetes API internals
• controller development
• Operator SDK
• CustomResourceDefinitions
• ValidatingAdmissionPolicy
• policy-as-code with Kyverno, Gatekeeper, or validating admission policies
• Service Mesh with Istio, Linkerd, or Consul
• eBPF and Cilium
• Cluster API
• multi-cluster GitOps
• progressive delivery with Argo Rollouts or Flagger
• canary, blue-green, and A/B deployments
• Workload Identity
• Confidential Containers
• sandbox runtimes
• GPU and accelerator scheduling
• Dynamic Resource Allocation
• backup solutions such as Velero
• Kubernetes cost management
• FinOps for platform teams
• SLO engineering on Kubernetes
• chaos engineering

22.1. Deep-Dive Module: Controllers and Operators

Learning objectives:

• You understand how controllers reconcile desired states.
• You can classify CRDs as API extensions.
• You recognize risks from poorly operated operators.

Content:

• Reconciliation Loop
• Owner References
• Finalizers
• Conditions
• CRDs
• Controller Runtime
• Operator pattern
• upgrade and backup questions for CRDs

22.2. Deep-Dive Module: Policy-as-Code

Learning objectives:

• You can validate technical platform rules automatically.
• You understand the difference between validation, mutation, and audit.

Content:

• Admission Control
• ValidatingAdmissionPolicy
• Kyverno
• Gatekeeper
• policy tests
• exception processes
• CI/CD policy checks

22.3. Deep-Dive Module: Progressive Delivery

Learning objectives:

• You can reduce release risk.
• You understand rollout strategies beyond the standard rolling update.

Content:

• canary
• blue-green
• feature flags
• traffic splitting
• metrics-based promotion
• automatic rollbacks
• Argo Rollouts
• Flagger

22.4. Deep-Dive Module: Kubernetes for Regulated Environments

Learning objectives:

• You can translate audit, compliance, and traceability requirements into Kubernetes practices.

Content:

• audit logs
• RBAC evidence
• policy reports
• image provenance
• Secret rotation
• encryption
• tenant separation
• backup evidence
• change management

23. Sources and References

The following references are useful for deeper study and for keeping the curriculum current:

• Kubernetes Documentation: https://kubernetes.io/docs/
• Kubernetes Concepts: https://kubernetes.io/docs/concepts/
• Kubernetes Components: https://kubernetes.io/docs/concepts/overview/components/
• Kubernetes API Concepts: https://kubernetes.io/docs/reference/using-api/api-concepts/
• Pods: https://kubernetes.io/docs/concepts/workloads/pods/
• Deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
• Services: https://kubernetes.io/docs/concepts/services-networking/service/
• Gateway API: https://kubernetes.io/docs/concepts/services-networking/gateway/
• NetworkPolicies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• PersistentVolumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• Resource Management: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
• Probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
• RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
• Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
• kubectl Reference: https://kubernetes.io/docs/reference/kubectl/
• kubeadm Reference: https://kubernetes.io/docs/reference/setup-tools/kubeadm/
• Creating a Cluster with kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• kubeadm Upgrade: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
• etcd Disaster Recovery: https://etcd.io/docs/
• Helm Documentation: https://helm.sh/docs/
• Kustomize Documentation: https://kubectl.docs.kubernetes.io/references/kustomize/
• Argo CD Documentation: https://argo-cd.readthedocs.io/
• Flux Documentation: https://fluxcd.io/flux/
• Cilium Documentation: https://docs.cilium.io/
• Falco Documentation: https://falco.org/docs/
• OpenTelemetry Documentation: https://opentelemetry.io/docs/

Appendix A: Short Matrix of Core Resources

Appendix B: Minimum Standard for Workshop Manifests

Whenever possible, every manifest in the workshop should include:

• clear names,
• Namespace,
• labels,
• defined selectors,
• resource specifications,
• probes for long-running applications,
• SecurityContext where applicable,
• no real Secrets,
• traceable image versions,
• comments only where they improve understanding.

Example structure:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: workshop
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: kubernetes-workshop
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
    spec:
      serviceAccountName: web
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 20
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              memory: "256Mi"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

Appendix C: Teardown Checklist

Before the end of an exercise, you should check:

• Which Namespaces were created?
• Which Deployments, Services, Jobs, and Pods are still running?
• Which PVCs and PVs remain?
• Which LoadBalancers were created?
• Which Secrets contain test data?
• Which RBAC bindings were created?
• Which NetworkPolicies affect subsequent exercises?
• Which Helm releases are installed?

Example commands:

kubectl get all -n workshop
kubectl get configmap,secret,pvc,ingress,networkpolicy -n workshop
helm list -n workshop
kubectl delete namespace workshop

Appendix D: Final Review Questions

• Which resource represented the authoritative desired state?
• Which controllers were involved?
• Which conditions or events were relevant for diagnosis?
• Which change was risky?
• How could you have validated the change beforehand?
• How was success measured?
• How is the solution torn down?
• Which security assumption is still open?
• Which operational task needs to be automated?
• Which documentation is missing for production operations?