Kubernetes Workshop: Professional – Exercises with Solutions
Reference: 03-professional-EN.md
Difficulty level: Professional
Format: architecture, operations, and hands-on walkthrough with sample solutions
Goal of this exercise document
This exercise document complements the Professional workshop with production-oriented tasks. The focus is on platform blueprint, operating model, managed/self-managed/hybrid, Node Pools, cluster sizing, upgrades, API deprecations, observability, autoscaling, GitOps, multi-tenancy, operators, CRDs, backup, disaster recovery, cost management, and runbooks.
Professional means here: You prove not only that an application runs. You define how a platform for multiple teams is operated, upgraded, observed, scaled, restored, and managed with cost awareness.
Prerequisites
You need:
- a lab cluster or simulated cluster contexts
kubectl- optional Helm and Kustomize
- optional Metrics Server, Prometheus/Grafana, or a provided observability environment
- optional GitOps controllers such as Argo CD or Flux
- optional Velero or a backup demo
Recommended working directory:
kubernetes/professional-exercises/
platform-blueprint.md
upgrade-runbook.md
observability-slo.md
autoscaling-review.md
gitops-flow.md
tenant-template/
operator-review.md
dr-runbook.md
cost-review.md
Exercise overview
| No. | Topic | Result |
|---|---|---|
| 1 | Platform blueprint | production-oriented cluster standard |
| 2 | Operating model | Managed, self-managed, hybrid evaluated |
| 3 | Node Pools and sizing | capacity and pool decision |
| 4 | Upgrade runbook | minor upgrade with deprecation check |
| 5 | Observability and SLO | metrics, logs, Events, alerts, runbook link |
| 6 | Autoscaling | HPA, VPA, Cluster Autoscaler, Karpenter, KEDA evaluated |
| 7 | GitOps and CI/CD | clean promotion and rollback flow |
| 8 | Multi-tenant Namespace | Namespace template with governance |
| 9 | Operators and CRDs | operations review for API extension |
| 10 | Backup and DR | restore scenarios for Namespace and cluster |
| 11 | Cost management | rightsizing and showback note |
| 12 | Operations handover | runbook, stop criteria, and evidence |
Part A – Exercises
Exercise 1: Create a platform blueprint
Goal
You describe a repeatable standard for a production-oriented Kubernetes cluster.
Task
Create platform-blueprint.md with these sections:
- Platform goal
- Cluster model: shared cluster or cluster per environment
- Environment separation: dev, test, stage, prod
- Responsibilities: platform team, app team, security, operations
- Standard add-ons: CNI, CSI, CoreDNS, Ingress/Gateway, metrics, observability, GitOps, backup
- Namespace standard: Labels, owner, RBAC, Quotas, LimitRanges, NetworkPolicy, Pod Security
- Upgrade and deprecation process
- Observability and SLO model
- Backup and restore model
- Cost model: showback or chargeback
Expected result
A second person can derive from the blueprint how a new cluster should be built and operated.
Exercise 2: Evaluate managed, self-managed, and hybrid
Goal
You make a defensible platform decision and document trade-offs.
Task
Create a decision matrix for three scenarios:
- Managed Kubernetes for standard web applications
- Self-managed Kubernetes for highly specialized bare-metal workloads
- Hybrid model for cloud and on-premises components
Evaluate:
- control-plane responsibility,
- upgrade effort,
- security responsibility,
- cost and provider dependency,
- operational risk,
- required team skills.
Exercise 3: Plan Node Pools and cluster sizing
Goal
You plan Node Pools, capacity buffer, and add-on resources for a production-oriented platform.
Scenario
The platform must host the following workloads:
- 20 standard web services
- 5 CPU-intensive workers
- 3 stateful services with PVC
- observability stack
- Ingress/Gateway Controller
- GitOps Controller
- backup controller
Task
- Design at least three Node Pools.
- Plan system and add-on overhead.
- Define capacity buffers for rollouts and Node failures.
- Document cloud provider quotas that must be checked.
- Write stop criteria for scaling.
Exercise 4: Write an upgrade runbook with API deprecation check
Goal
You plan a Kubernetes minor upgrade as a change with validation and rollback path.
Task
Create upgrade-runbook.md with:
- target version and current version
- version skew check
- API deprecation check
- CRD and webhook check
- add-on compatibility
- backup and restore point
- test cluster or staging
- control-plane upgrade
- add-on check
- Node Pool upgrade with cordon and drain
- smoke tests
- stop criteria
- rollback or recovery plan
- final report
Exercise 5: Define observability model and SLO
Goal
You describe observability not as a tool list, but as an operating model with signals, SLOs, alerts, and runbooks.
Task
Define for a sample app checkout-api:
- Service Level Objective,
- Service Level Indicator,
- metrics,
- logs,
- traces,
- Kubernetes Events,
- dashboards,
- alerts,
- runbook link,
- retention.
Also add platform signals for API Server, CoreDNS, Ingress, Nodes, and PVCs.
Exercise 6: Evaluate autoscaling
Goal
You classify HPA, VPA, Cluster Autoscaler, Karpenter, and KEDA by purpose, prerequisites, and risk.
Scenario
A queue-based worker application processes irregular jobs. Load spikes occur during the day. At night, the platform should save costs.
Task
- Define CPU and memory requests.
- Design an HPA strategy.
- Decide whether KEDA is useful.
- Describe interaction with Node autoscaling.
- Formulate risks such as flapping, cost increase, and missing metrics.
Exercise 7: Sketch GitOps and CI/CD flow
Goal
You separate CI and GitOps cleanly and avoid permanent broad cluster rights in the CI pipeline.
Task
Create gitops-flow.md with a flow of:
- code commit
- CI: build, test, scan
- image push to registry
- digest determination
- pull request for manifest change
- review and approval
- GitOps sync
- drift detection
- rollback by Git revert
Document which identity needs which rights.
Exercise 8: Create a multi-tenant Namespace template
Goal
You create a Namespace pattern for a team that can deploy in a limited way without receiving cluster-wide rights.
Task
Create manifests for:
- Namespace with Labels,
- ResourceQuota,
- LimitRange,
- Role,
- RoleBinding,
- NetworkPolicy Default Deny,
- Pod Security Labels,
- owner annotation.
Document which controls are only prepared at Professional level and enforced more strictly at Advanced level.
Exercise 9: Evaluate Operator and CRD
Goal
You evaluate whether an Operator is operationally viable.
Task
Choose a hypothetical Operator database-operator and document:
- installed CRDs,
- required ClusterRoles,
- managed Custom Resources,
- Conditions and Events,
- backup relevance,
- upgrade risks,
- uninstall risks,
- multi-tenancy capability,
- stop criteria for production.
Exercise 10: Plan backup and disaster recovery
Goal
You distinguish Git state, etcd, PVC data, Secrets, CRDs, and external systems.
Task
Create dr-runbook.md with two scenarios:
- restore of a single Namespace.
- restore of an entire cluster.
For each scenario, describe:
- affected data classes,
- required backups,
- restore order,
- validation,
- RTO/RPO assumption,
- risks,
- evidence.
Exercise 11: Perform cost analysis and rightsizing
Goal
You analyze cost factors and formulate technical rightsizing actions.
Scenario
A cluster shows:
- many Pods with high requests and low usage,
- several unused PVCs,
- several external Load Balancers,
- continuously running dev workloads,
- no showback assignment per Namespace.
Task
Create cost-review.md with:
- cost sources,
- technical measurement points,
- rightsizing recommendations,
- risks of underrequesting,
- showback model,
- action plan.
Exercise 12: Prepare operations handover
Goal
You create an operations handover that another person can use in on-call or operations context.
Task
Create a short handover with:
- platform overview,
- critical add-ons,
- known SLOs,
- alerting paths,
- upgrade windows,
- backup and restore responsibility,
- top 5 runbooks,
- stop criteria,
- open risks.
Part B – Sample solutions and walkthroughs
Solution 1: Platform blueprint
Sample structure:
# Platform Blueprint
## 1. Goal
The platform provides standardized Kubernetes environments for multiple application teams. It offers secure Namespaces, GitOps-based deployment, central observability, Ingress/Gateway, storage, backup, and documented upgrade processes.
## 2. Cluster model
- Dev/test: shared cluster with Namespace isolation.
- Stage/prod: separate clusters by criticality or environment.
- Prod clusters across at least three Availability Zones.
## 3. Responsibilities
| Area | Owner |
| --- | --- |
| Control Plane | provider or platform team |
| Nodes and Node Pools | platform team |
| CNI/CSI/CoreDNS | platform team |
| Namespace contents | application team |
| Security Policies | platform team + security |
| App SLOs | application team |
| Platform SLOs | platform team |
## 4. Standard add-ons
| Add-on | Purpose | Owner | SLO |
| --- | --- | --- | --- |
| CNI | Pod network and NetworkPolicy | platform | Pod network available |
| CSI | PV provisioning | platform | PVC provisioning stable |
| CoreDNS | service discovery | platform | DNS latency and error rate |
| Ingress/Gateway | external HTTP access | platform | 99.9% routing availability |
| Observability | metrics/logs/traces/Events | platform | signal availability |
| GitOps | desired-state sync | platform | reliable sync |
| Backup | restore capability | platform | restore tested |
## 5. Namespace standard
Every tenant Namespace receives:
- owner Labels,
- ResourceQuota,
- LimitRange,
- Role/RoleBinding,
- Default-Deny NetworkPolicy,
- Pod Security Labels,
- cost Label.
## 6. Lifecycle
- monthly deprecation scan,
- quarterly Kubernetes minor-upgrade window,
- staging test before prod,
- documented stop criteria,
- smoke tests after every upgrade step.
Assessment:
A good blueprint is not tool-fixed. It describes owners, SLOs, upgrade paths, isolation, costs, and recovery.
Solution 2: Managed, self-managed, and hybrid
Sample decision:
| Criterion | Managed | Self-managed | Hybrid |
|---|---|---|---|
| Control Plane | provider | internal team | shared |
| Upgrade effort | lower | high | medium to high |
| Control | limited | maximum | differentiated |
| Security responsibility | shared | mostly internal | shared and complex |
| Provider dependency | high | lower | medium |
| Operational risk | provider boundaries | operational errors | integration errors |
| Team skills | basic Kubernetes operations | deep Kubernetes/etcd/PKI knowledge | cloud + on-prem + networking |
Sample assessment:
- Managed Kubernetes is the default for most web applications if provider boundaries are acceptable.
- Self-managed is useful only when special requirements justify the additional complexity.
- Hybrid is not a cost-free compromise; it increases governance, network, and operational effort.
Solution 3: Node Pools and sizing
Sample Node Pools:
| Node Pool | Purpose | Characteristics |
|---|---|---|
| system-pool | CoreDNS, CNI, CSI, Ingress, Observability Agents | Taints for system add-ons, high stability |
| app-general | standard web services | horizontal scaling, distributed across zones |
| worker-cpu | CPU-intensive workers | larger CPU, possibly spot with PDB/retry |
| stateful | stateful services with PVC | storage-near Nodes, no aggressive spot strategies |
Sizing checklist:
- Sum of app requests plus DaemonSet overhead.
- Space for rolling updates: at least additional capacity for new Pods during rollout.
- Space for zone failure or Node failure, depending on SLA.
- Do not run CoreDNS, logging, metrics, and Ingress with minimal values from small clusters.
- Check cloud quotas for compute, IPs, volumes, Load Balancers, firewall rules, and egress.
Stop criteria:
- Cluster Autoscaler can no longer add Nodes.
- IP address space exhausted.
- CoreDNS or CNI shows high error rate.
- Pods remain
Pendingbecause of resources. - Add-ons run into OOMKill or CPU throttling.
Solution 4: Upgrade runbook
Sample content:
# Upgrade Runbook
## Goal
Upgrade from Kubernetes v1.x to v1.y.
## Pre-check
- kubectl version
- kubectl get nodes -o wide
- kubectl get pods -A
- kubectl get apiservices
- kubectl get crds
- kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
## API deprecation check
- Scan Git repositories for old API versions.
- Render and check Helm charts.
- Render and check Kustomize overlays.
- Check CRD versions and conversion webhooks.
## Backup
- etcd backup or provider snapshot.
- Velero/Namespace backup, if available.
- PV snapshots for stateful workloads.
- Record GitOps repository revision.
## Sequence
1. Announce change.
2. Test staging.
3. Upgrade Control Plane.
4. Validate add-ons.
5. Upgrade Node Pools one by one.
6. Per Node: cordon, drain, upgrade, uncordon.
7. Smoke tests.
8. Final report.
## Stop criteria
- API Server error rate increased.
- CoreDNS not stable.
- CNI/CSI incompatible.
- critical workloads not Ready.
- webhook blocks deployments.
- deprecation finding not fixed.
Example commands:
kubectl api-resources
kubectl api-versions
kubectl get apiservices
kubectl get crds
kubectl get events -A --sort-by=.metadata.creationTimestamp
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>
Solution 5: Observability and SLO
Sample SLO:
# Observability SLO checkout-api
## SLO
99.9% of HTTP requests to `/checkout` are successful within 300 ms over a 30-day window.
## SLI
- Success rate: HTTP 2xx/3xx divided by all requests, excluding explicitly excluded 4xx.
- Latency: p95 request duration.
## Signals
| Signal | Source | Purpose |
| --- | --- | --- |
| Request Rate | Ingress/app metrics | load profile |
| Error Rate | app/Ingress metrics | availability |
| Duration p95/p99 | app/Ingress metrics | latency |
| Logs | app logs | root-cause analysis |
| Traces | OpenTelemetry | dependency analysis |
| Events | Kubernetes API | scheduling, probe, pull, PVC |
| Resource Usage | cAdvisor/kubelet | CPU/memory/throttling |
## Alerts
- Error Rate > 2% for 10 minutes.
- p95 > 300 ms for 15 minutes.
- Deployment not fully available for 10 minutes.
- HPA at maximum for 15 minutes.
## Runbooks
- checkout-api-high-error-rate.md
- checkout-api-latency.md
- checkout-api-rollout-failure.md
Platform signals:
| Platform component | Metrics/signals |
|---|---|
| API Server | latency, error rate, request rate |
| CoreDNS | DNS errors, latency, replicas |
| Ingress/Gateway | 5xx, latency, backend health |
| Nodes | Ready, MemoryPressure, DiskPressure, PIDPressure |
| PVC/CSI | Pending PVCs, attach/detach errors |
Solution 6: Autoscaling
Sample assessment:
| Mechanism | Use in scenario | Prerequisite | Risk |
|---|---|---|---|
| HPA | scales worker Pods by CPU or custom metric | Metrics API, requests | wrong metric, flapping |
| VPA | recommends requests for workers | VPA Recommender | conflict with HPA on CPU |
| Cluster Autoscaler | adds Nodes for Pending Pods | cloud/Node-group integration | slow reaction, quotas |
| Karpenter | flexible Node provisioning | supported cloud, policies | cost control, NodeClass design |
| KEDA | scales by queue length | scaler and external metric | external metric availability |
Sample HPA:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: worker
namespace: worker-prod
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: worker
minReplicas: 2
maxReplicas: 30
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
behavior:
scaleDown:
stabilizationWindowSeconds: 300
Sample decision:
For queue-based load, KEDA is often technically better than pure CPU scaling because queue length represents the work backlog more directly. HPA can still be useful as a supplement when CPU per job varies significantly.
Solution 7: GitOps and CI/CD
Sample flow:
Code Commit
-> CI builds image
-> CI runs tests and scan
-> CI pushes image to registry
-> CI determines image digest
-> CI creates pull request in the manifest repository
-> Review checks manifest, policy, and diff
-> Merge updates desired state
-> GitOps Controller synchronizes cluster
-> Drift Detection reports deviations
-> Rollback happens through Git Revert
Rights:
| Identity | Rights |
|---|---|
| CI Build Identity | registry push, scan, create PR |
| Reviewer | Git review and approval |
| GitOps Controller | limited cluster rights on target Namespaces |
| App team | Namespace-specific deploy rights or PR rights |
| Platform team | controllers, policies, cluster resources |
Stop criterion:
The CI pipeline receives no permanent cluster-admin rights. Cluster state is synchronized through GitOps, not through imperative CI deployments.
Solution 8: Multi-tenant Namespace
Namespace:
apiVersion: v1
kind: Namespace
metadata:
name: team-a
labels:
owner: team-a
environment: prod
cost-center: cc-1234
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
annotations:
platform.example.com/owner-email: team-a@example.com
ResourceQuota:
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-a-quota
namespace: team-a
spec:
hard:
pods: "20"
requests.cpu: "4"
requests.memory: 8Gi
limits.cpu: "8"
limits.memory: 16Gi
persistentvolumeclaims: "5"
LimitRange:
apiVersion: v1
kind: LimitRange
metadata:
name: team-a-defaults
namespace: team-a
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 128Mi
default:
cpu: 500m
memory: 512Mi
Role and RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: app-deployer
namespace: team-a
rules:
- apiGroups: ["", "apps", "batch", "networking.k8s.io"]
resources: ["pods", "services", "configmaps", "deployments", "replicasets", "jobs", "cronjobs", "ingresses"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-deployer-binding
namespace: team-a
subjects:
- kind: Group
name: team-a-developers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: app-deployer
apiGroup: rbac.authorization.k8s.io
Default-Deny NetworkPolicy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: team-a
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Professional boundary:
- Professional models tenant governance.
- Advanced + Security checks hard RBAC escalation paths, Admission Enforcement, Secrets, threat model, and incident response.
Solution 9: Operator review
Sample review:
# Operator Review: database-operator
## Purpose
Automates database instances through Custom Resources.
## CRDs
- databases.example.com
- databasebackups.example.com
- databaserestores.example.com
## Rights
The Operator requires access to StatefulSets, Services, Secrets, PVCs, Events, and its own CRDs. Cluster-wide rights must be justified.
## Conditions
- Ready
- Progressing
- Degraded
- BackupReady
- RestoreFailed
## Backup
Custom Resources alone are not enough. PVC data, Secrets, and external backups must be included.
## Upgrade risks
- CRD schema changes
- conversion webhooks
- finalizers block deletion
- Operator image not compatible
- old Custom Resources cannot be migrated
## Stop criteria
- unclear ClusterRole with `*`
- no documented restore test
- no CRD versioning strategy
- no uninstall guide
- no multi-tenancy capability
Solution 10: Backup and DR
Namespace restore:
# Namespace Restore
## Data classes
- manifests from Git
- Secrets from secret management
- PVC data from VolumeSnapshot or backup
- CRDs, if the Namespace contains Custom Resources
## Order
1. Create target Namespace.
2. Apply Quotas, RBAC, and Policies.
3. Restore Secrets.
4. Restore PVCs or VolumeSnapshots.
5. Synchronize workloads from GitOps.
6. Check Service and Ingress/Gateway paths.
7. Run smoke tests.
## Evidence
- restore timestamp
- Git revision
- backup ID
- successful smoke tests
- open deviations
Cluster restore:
# Cluster Restore
## Data classes
- cluster add-ons
- CRDs
- Namespaces
- RBAC
- Secrets
- PV/PVC data
- external systems: DNS, registry, identity provider
## Order
1. provision a new cluster or perform etcd restore.
2. validate CNI, CSI, CoreDNS.
3. install CRDs.
4. install GitOps Controller.
5. synchronize platform Namespaces.
6. synchronize tenant Namespaces.
7. restore PV data.
8. validate Ingress/DNS.
9. run SLO smoke tests.
Important statement:
A backup is reliable only after a restore test has been performed and documented.
Solution 11: Cost analysis and rightsizing
Sample analysis:
| Cost source | Measurement point | Action |
|---|---|---|
| CPU/memory overrequesting | requests vs. usage | adjust requests based on p95/p99 |
| idle workloads | replicas without traffic | dev time windows, check scale-to-zero |
| PVCs | PVC without Bound Pod | check owner, snapshot, deletion |
| Load Balancers | Services of type LoadBalancer | use shared Ingress/Gateway |
| Egress | cloud billing and flow logs | optimize data paths |
| Spot/on-demand | Node Pool costs | check spot for fault-tolerant workers |
Sample showback:
Every Namespace needs Labels:
metadata:
labels:
owner: team-a
cost-center: cc-1234
environment: prod
Risk of underrequesting:
Requests that are too low appear to save costs, but they can degrade scheduling, HPA, Node utilization, and stability. Rightsizing does not mean “set requests to the minimum”; it means defining realistic operating values.
Solution 12: Operations handover
Sample handover:
# Kubernetes Platform Operations Handover
## Overview
- Cluster: prod-eu-1
- Model: Managed Kubernetes
- Environments: stage/prod separated
- GitOps: Argo CD
- Ingress: Gateway Controller
- Observability: Prometheus, Grafana, Loki, OpenTelemetry
## Critical add-ons
| Add-on | Owner | Runbook |
| --- | --- | --- |
| CoreDNS | platform | runbooks/coredns.md |
| CNI | platform | runbooks/network.md |
| CSI | platform | runbooks/storage.md |
| GitOps | platform | runbooks/gitops.md |
| Ingress/Gateway | platform | runbooks/ingress.md |
## Stop criteria
- API Server unreachable.
- CoreDNS error rate critical.
- CNI blocks Pod networking.
- CSI cannot attach volumes.
- GitOps synchronizes the wrong desired state.
## Open risks
- Last cluster restore test was more than 90 days ago.
- Some Namespaces have no cost Label.
- Two old API versions found in legacy chart.
Assessment checklist
| Criterion | Completed? |
|---|---|
| Platform blueprint complete | ☐ |
| Operating model justified | ☐ |
| Node Pool and sizing decision traceable | ☐ |
| Upgrade runbook with stop criteria | ☐ |
| Deprecation check included | ☐ |
| Observability with SLO and runbooks | ☐ |
| Autoscaling evaluated by metric dependencies | ☐ |
| GitOps and CI/CD cleanly separated | ☐ |
| Multi-tenant Namespace modeled | ☐ |
| Operator review with CRD/RBAC/backup risks | ☐ |
| DR scenarios with restore order | ☐ |
| Cost analysis with rightsizing and showback | ☐ |