Diagrams: Kubernetes
This document is a visual reference for your Kubernetes workshop. It complements the concepts, architecture, best-practices, command-reference, and curriculum documents with explanatory diagrams, sequence views, and troubleshooting flows.
The diagrams are intentionally designed for teaching. They do not show every implementation detail. Instead, they focus on the relationships, responsibilities, and decision paths that you should be able to explain, discuss, and reproduce in practical workshop exercises.
1. Usage Notes
1.1 Purpose of this diagram file
Use this file to understand Kubernetes visually, not only through text. Each diagram answers a typical workshop question:
- Which component is responsible for what?
- Which resource creates or controls which other resource?
- Where is Desired State stored?
- Where does a failure mode originate?
- Which point do you inspect first?
- Which responsibility belongs to Kubernetes, to an add-on, to the cloud provider, or to you?
The diagrams are suitable for presentations, whiteboard explanations, group exercises, and troubleshooting labs.
1.2 Diagram types
| Diagram type | Purpose | Typical use |
|---|---|---|
flowchart |
Components, data flows, dependencies | Architecture, networking, storage, security |
sequenceDiagram |
Time-ordered flows | API request, rollout, DNS, admission, kubeadm |
stateDiagram-v2 |
States and transitions | Pod lifecycle, rollout, incident response |
| Decision tree | Analysis paths | Troubleshooting and design decisions |
1.3 Scope and limits of representation
Kubernetes is extensible. A diagram can therefore look different depending on the distribution. Managed Kubernetes, kubeadm clusters, OpenShift, k3s, Talos, Cilium, Calico, Istio, Argo CD, or a cloud load balancer all change implementation details. The core logic remains the same: API, desired state, controllers, scheduler, kubelet, runtime, networking, storage, security, and observability work together.
2. Big Picture and Cluster Architecture
2.1 End-to-end overview: From you to a running application
This diagram shows the high-level context. You define a desired state, Kubernetes stores it, and multiple controllers plus node components work to align the actual state with it.
flowchart LR
user["You / Team"] --> intent["Desired State"]
intent --> manifest["YAML, Helm, Kustomize, or GitOps"]
manifest --> api["Kubernetes API Server"]
api --> etcd["etcd: Cluster State"]
api --> controllers["Controllers"]
api --> scheduler["Scheduler"]
scheduler --> node["Worker Node"]
controllers --> node
node --> kubelet["kubelet"]
kubelet --> runtime["Container Runtime"]
runtime --> pods["Pods and Containers"]
pods --> service["Service Discovery and Networking"]
pods --> storage["Volumes and Storage"]
pods --> signals["Logs, Metrics, Events"]
signals --> user
Rule of thumb: Kubernetes does not merely start containers. Kubernetes manages desired state through an API and continuously corrects drift.
2.2 Control Plane and Worker Nodes
The Control Plane decides and coordinates. Worker Nodes run workloads. The API Server is the central interface; direct changes on nodes bypass the Kubernetes model and are usually not reproducible.
flowchart TD
subgraph cp["Control Plane"]
api["kube-apiserver"]
etcd["etcd"]
sched["kube-scheduler"]
cm["kube-controller-manager"]
ccm["cloud-controller-manager"]
end
subgraph n1["Worker Node A"]
kubelet1["kubelet"]
proxy1["kube-proxy or eBPF dataplane"]
cni1["CNI Plugin"]
rt1["containerd or CRI-O"]
pods1["Pods"]
end
subgraph n2["Worker Node B"]
kubelet2["kubelet"]
proxy2["kube-proxy or eBPF dataplane"]
cni2["CNI Plugin"]
rt2["containerd or CRI-O"]
pods2["Pods"]
end
api --> etcd
api --> sched
api --> cm
api --> ccm
api --> kubelet1
api --> kubelet2
kubelet1 --> rt1 --> pods1
kubelet2 --> rt2 --> pods2
cni1 --> pods1
cni2 --> pods2
proxy1 --> pods1
proxy2 --> pods2
Workshop note: Ask participants which component actually starts a Pod. The correct answer is not the Scheduler; it is kubelet through the Container Runtime. The Scheduler only assigns an appropriate Node.
2.3 Highly available Control Plane
In production-like environments, you run the Control Plane redundantly. The API Server can be scaled horizontally. etcd requires quorum and should be especially protected, backed up, and monitored.
flowchart TD
client["kubectl, Controllers, CI/CD, GitOps"] --> lb["Load Balancer for API Servers"]
lb --> api1["API Server 1"]
lb --> api2["API Server 2"]
lb --> api3["API Server 3"]
api1 --> e1["etcd Member 1"]
api2 --> e2["etcd Member 2"]
api3 --> e3["etcd Member 3"]
e1 <--> e2
e2 <--> e3
e1 <--> e3
api1 --> sched["Scheduler with Leader Election"]
api2 --> cm["Controller Manager with Leader Election"]
api3 --> ccm["Cloud Controller Manager"]
Rule of thumb: API Server redundancy protects the control interface. etcd quorum protects cluster state. Without a tested etcd backup, you do not have a dependable disaster-recovery path.
2.4 Stacked etcd vs. external etcd
For kubeadm-based and self-managed clusters, the etcd topology is an important architecture decision. Stacked etcd is more compact. External etcd separates responsibilities more clearly.
flowchart LR
subgraph stacked["Stacked Control Plane"]
cp1["Control Plane Node 1\nAPI + etcd"]
cp2["Control Plane Node 2\nAPI + etcd"]
cp3["Control Plane Node 3\nAPI + etcd"]
cp1 <--> cp2
cp2 <--> cp3
cp1 <--> cp3
end
subgraph external["External etcd topology"]
apiA["Control Plane Nodes\nAPI, Scheduler, Controller"]
etcdA["dedicated etcd Node 1"]
etcdB["dedicated etcd Node 2"]
etcdC["dedicated etcd Node 3"]
apiA --> etcdA
apiA --> etcdB
apiA --> etcdC
etcdA <--> etcdB
etcdB <--> etcdC
etcdA <--> etcdC
end
Decision aid: Use stacked etcd for smaller, simpler setups. Use external etcd when you need stronger separation, dedicated operational processes, and clearer ownership.
2.5 Inside a Worker Node
A Node is not just a VM or server. Kubernetes expects kubelet, a runtime, networking components, Pod sandboxes, local logs, and often additional DaemonSets.
flowchart TD
api["API Server"] --> kubelet["kubelet"]
subgraph node["Worker Node"]
kubelet --> cri["CRI interface"]
cri --> runtime["containerd or CRI-O"]
runtime --> sandbox["Pod Sandbox"]
sandbox --> c1["Container app"]
sandbox --> c2["Container sidecar"]
kubelet --> volumes["Volume Mounts"]
kubelet --> probes["Liveness, Readiness, Startup Probes"]
kubelet --> logs["Container Logs"]
proxy["kube-proxy or eBPF"] --> dataplane["Service dataplane"]
cni["CNI Plugin"] --> podnet["Pod network"]
ds["DaemonSet Pods"] --> nodeagents["Monitoring, Logging, Security, CNI"]
end
Inspection question: If a Pod does not start, you do not inspect only the manifest. You also inspect Node state, kubelet, runtime, CNI, image pull, volumes, resources, and events.
2.6 Control-Plane-to-Node communication
The API Server talks to kubelet and can use proxy functionality to reach Node, Pod, or Service targets. Many debug commands work only when these paths are reachable and authorized.
flowchart LR
api["API Server"] -->|Read Pod status| kubelet["kubelet"]
api -->|Logs, exec, port-forward, debug| kubelet
kubelet -->|Report Node and Pod status| api
api -->|Proxy path| target["Node, Pod, or Service"]
target --> response["Response to API client"]
response --> user["kubectl"]
Troubleshooting note: If kubectl logs, kubectl exec, or kubectl port-forward fails, the issue can be API access, RBAC, the kubelet port, Node networking, or kubelet itself.
2.7 Kubernetes components by responsibility area
Not every component solves every problem. This diagram helps map failure modes to a layer faster.
flowchart TD
platform["Platform"] --> api["API and State"]
platform --> scheduling["Scheduling"]
platform --> runtime["Runtime"]
platform --> network["Networking"]
platform --> storage["Storage"]
platform --> security["Security"]
platform --> observe["Observability"]
api --> apiserver["kube-apiserver"]
api --> etcd["etcd"]
scheduling --> scheduler["kube-scheduler"]
scheduling --> autoscaler["Cluster Autoscaler"]
runtime --> kubelet["kubelet"]
runtime --> crio["CRI Runtime"]
network --> cni["CNI"]
network --> service["Service dataplane"]
storage --> csi["CSI"]
storage --> pv["PV, PVC, StorageClass"]
security --> rbac["RBAC"]
security --> admission["Admission"]
security --> pss["Pod Security"]
observe --> metrics["Metrics"]
observe --> logs["Logs"]
observe --> traces["Traces"]
3. API, Resource Model, and Reconciliation
3.1 API request lifecycle
Every change goes through the API Server. Authentication, authorization, admission, and persistence are separate steps. This separation is central for security and troubleshooting.
sequenceDiagram
participant U as You or Tool
participant API as API Server
participant AUTHN as Authentication
participant AUTHZ as Authorization
participant ADM as Admission Control
participant ETCD as etcd
participant CTRL as Controller
U->>API: apply, create, patch, or delete
API->>AUTHN: Who is making the request?
AUTHN-->>API: Identity
API->>AUTHZ: Is this identity allowed to do this?
AUTHZ-->>API: allowed or denied
API->>ADM: Validation and mutation
ADM-->>API: accepted or rejected
API->>ETCD: Store state
ETCD-->>API: stored
API-->>U: API response
CTRL->>API: Watch change
CTRL->>API: Adjust actual state
Rule of thumb: A kubectl apply takes effect only after the API Server accepts the request and stores the desired state. Execution then happens asynchronously through controllers and kubelet.
3.2 Anatomy of a Kubernetes object
Almost every Kubernetes object follows the same mental model: type, metadata, desired state, and observed state.
flowchart TD
obj["Kubernetes Object"] --> apiVersion["apiVersion"]
obj --> kind["kind"]
obj --> metadata["metadata"]
obj --> spec["spec"]
obj --> status["status"]
metadata --> name["name"]
metadata --> namespace["namespace"]
metadata --> labels["labels"]
metadata --> annotations["annotations"]
metadata --> owner["ownerReferences"]
metadata --> finalizers["finalizers"]
spec --> desired["desired state"]
status --> actual["observed state"]
status --> conditions["Conditions"]
status --> observed["observedGeneration"]
Workshop note: Have participants inspect kubectl get deployment demo -o yaml and find spec, status, metadata.generation, and status.observedGeneration.
3.3 Desired State, Actual State, and Reconciliation
Controllers run in loops. They watch resources, compare desired and actual state, perform actions, and write status back.
flowchart LR
desired["Desired State\nin the API object"] --> compare["Controller compares"]
actual["Actual State\nin the cluster"] --> compare
compare -->|Drift| act["Perform action"]
act --> actual
compare -->|No drift| steady["stable state"]
act --> status["Update status"]
status --> desired
Rule of thumb: You do not primarily tell Kubernetes what to do once. You describe what the state should continuously look like.
3.4 Controller loop in detail
A controller is not a magical background process. It reacts to events, uses local caches, and attempts to correct missing or incorrect resources.
flowchart TD
watch["Watch API resources"] --> queue["Work Queue"]
queue --> reconcile["Reconcile function"]
reconcile --> read["Read current state"]
read --> decide{Is the actual state correct?}
decide -->|yes| status["Check or update status"]
decide -->|no| change["Create, update, or delete resources"]
change --> requeue["Requeue if needed"]
status --> wait["Wait for next change"]
requeue --> queue
wait --> watch
Operational relevance: If a controller is broken, API objects often still exist, but the actual state does not change as expected.
3.5 Owner References and Garbage Collection
Kubernetes uses Owner References to associate dependent resources. This allows Kubernetes to clean up downstream objects automatically.
flowchart TD
dep["Deployment"] -->|ownerReference| rs1["ReplicaSet old version"]
dep -->|ownerReference| rs2["ReplicaSet new version"]
rs2 -->|ownerReference| pod1["Pod 1"]
rs2 -->|ownerReference| pod2["Pod 2"]
rs1 --> oldpod["old Pod"]
delete["Deployment is deleted"] --> gc["Garbage Collector"]
gc --> rs1
gc --> rs2
gc --> pod1
gc --> pod2
Inspection question: Why should you not manually modify ReplicaSets or Pods of a Deployment as your primary operational path? Because the controller restores or replaces them from the Deployment state.
3.6 Finalizers and controlled deletion
Finalizers prevent an object from disappearing immediately. A controller can use them to clean up external resources before the object is finally removed.
sequenceDiagram
participant U as You
participant API as API Server
participant OBJ as Object with Finalizer
participant C as Controller
participant EXT as External Resource
U->>API: delete object
API->>OBJ: set deletionTimestamp
API-->>U: object is in deletion state
C->>API: sees object with deletionTimestamp
C->>EXT: clean up external resource
EXT-->>C: cleaned up
C->>API: remove finalizer
API->>API: delete object permanently
Troubleshooting note: If a Namespace, PVC, or Custom Resource stays in Terminating for a long time, inspect finalizers and the responsible controller.
3.7 Conditions, Status, and Events
Status and Events are your most important operational signals. They show what Kubernetes attempted and why something does not work.
flowchart LR
obj["Resource"] --> status["status"]
obj --> events["Events"]
status --> conditions["Conditions"]
conditions --> ready["Ready"]
conditions --> available["Available"]
conditions --> progressing["Progressing"]
conditions --> degraded["Degraded or Failed"]
events --> schedule["Scheduling Events"]
events --> image["Image Pull Events"]
events --> volume["Volume Events"]
events --> probe["Probe Events"]
Workshop command: kubectl describe is often the fastest entry point because it combines status, conditions, and events.
4. Workloads and Pod Lifecycle
4.1 Deployment, ReplicaSet, and Pods
A Deployment does not manage containers directly. It creates ReplicaSets, and ReplicaSets ensure the desired number of identical Pods.
flowchart TD
deploy["Deployment\napp=web"] --> rs["ReplicaSet\nreplicas=3"]
rs --> pod1["Pod web-1"]
rs --> pod2["Pod web-2"]
rs --> pod3["Pod web-3"]
svc["Service\nselector app=web"] --> pod1
svc --> pod2
svc --> pod3
Rule of thumb: The Deployment describes rollout strategy and Pod template. The ReplicaSet maintains the replica count. The Service finds matching Pods through labels.
4.2 Rolling update of a Deployment
During a rolling update, Kubernetes creates a new ReplicaSet and gradually scales down the old one. maxSurge and maxUnavailable determine how aggressively this transition happens.
sequenceDiagram
participant U as You
participant API as API Server
participant D as Deployment Controller
participant OLD as Old ReplicaSet
participant NEW as New ReplicaSet
participant SVC as Service
U->>API: new image through apply or set image
API-->>D: Deployment changed
D->>NEW: create new ReplicaSet
D->>NEW: scale up new Pods
NEW-->>SVC: readiness reached
D->>OLD: scale down old Pods
D->>API: update rollout status
Workshop note: During the update, show kubectl rollout status, kubectl get rs, and kubectl get pods -w side by side.
4.3 Rollback of a Deployment
A rollback uses stored rollout history. It is not a substitute for clean versioning, but it is an important operational mechanism.
flowchart LR
bad["Release v3 is faulty"] --> detect["Failure detected"]
detect --> undo["kubectl rollout undo"]
undo --> previous["scale ReplicaSet v2 back up"]
previous --> down["scale ReplicaSet v3 down"]
down --> verify["check Service, logs, and metrics"]
Inspection question: What happens if your ConfigMap or external database migration is not backward compatible? A Deployment rollback alone is then insufficient.
4.4 Pod lifecycle
A Pod moves through several states. Many failure modes become easier to understand if you treat phase, container state, and events separately.
stateDiagram-v2
[*] --> Pending
Pending --> Running: Node assigned and containers started
Pending --> Failed: Scheduling, image, or volume fails
Running --> Succeeded: all containers completed successfully
Running --> Failed: container or Pod fails permanently
Running --> Terminating: deletion or eviction
Terminating --> [*]
Succeeded --> [*]
Failed --> [*]
Important: CrashLoopBackOff is not a Pod phase. It is a container waiting reason. Inspect kubectl describe pod and kubectl logs --previous.
4.5 Init Container, App Container, and Sidecar pattern
Init Containers run before normal app containers. Sidecars complement an app at runtime, for example for proxying, logging, or synchronization.
flowchart TD
start["Pod is started"] --> init1["Init Container 1"]
init1 --> init2["Init Container 2"]
init2 --> app["App Container"]
init2 --> sidecar["Sidecar Container"]
app <--> sharednet["shared network namespace"]
sidecar <--> sharednet
app <--> volume["shared volumes"]
sidecar <--> volume
Design question: Does the function really belong into the same Pod, or would a separate Service be cleaner? A sidecar tightly couples lifecycle, resources, and scaling to the app.
4.6 Probes and traffic eligibility
Probes control self-healing and traffic routing. Readiness determines whether a Pod receives Service traffic. Liveness determines whether kubelet restarts a container. Startup protects slow starts.
flowchart TD
pod["Pod"] --> startup["Startup Probe"]
startup -->|successful| live["Liveness Probe active"]
startup -->|fails too often| restart1["Container restart"]
live -->|ok| ready["Readiness Probe"]
live -->|fails too often| restart2["Container restart"]
ready -->|ready| endpoints["EndpointSlice includes Pod"]
ready -->|not ready| noTraffic["no Service traffic"]
Anti-pattern: Do not use liveness as a substitute for readiness. A misconfigured liveness probe can push healthy but slow applications into restart loops.
4.7 Graceful shutdown
When a Pod terminates, traffic routing, signal handling, and terminationGracePeriodSeconds must work together.
sequenceDiagram
participant API as API Server
participant EP as EndpointSlice Controller
participant K as kubelet
participant APP as App Container
participant SVC as Service Traffic
API->>EP: Pod receives deletionTimestamp
EP->>SVC: remove Endpoint from routing
API->>K: terminate Pod
K->>APP: run preStop hook, if present
K->>APP: send SIGTERM
APP->>APP: finish in-flight requests
K->>APP: after grace period, SIGKILL if needed
Workshop note: Simulate a rollout with a short grace period and observe whether requests are interrupted.
4.8 StatefulSet and stable identity
StatefulSets are intended for workloads that need stable names, ordered startup, or stable volume association.
flowchart TD
sts["StatefulSet db"] --> pod0["db-0"]
sts --> pod1["db-1"]
sts --> pod2["db-2"]
pod0 --> pvc0["data-db-0"]
pod1 --> pvc1["data-db-1"]
pod2 --> pvc2["data-db-2"]
svc["Headless Service db"] --> pod0
svc --> pod1
svc --> pod2
dns0["db-0.db.namespace.svc"] --> pod0
dns1["db-1.db.namespace.svc"] --> pod1
dns2["db-2.db.namespace.svc"] --> pod2
Rule of thumb: A StatefulSet does not automatically make an application consistent. Replication, quorum, backup, and restore remain application or operator concerns.
4.9 DaemonSet for node-local functions
DaemonSets place one Pod on each selected Node. They are suitable for node agents, not for normal horizontal application scaling.
flowchart LR
ds["DaemonSet"] --> node1["Node A"]
ds --> node2["Node B"]
ds --> node3["Node C"]
node1 --> agent1["Agent Pod"]
node2 --> agent2["Agent Pod"]
node3 --> agent3["Agent Pod"]
agent1 --> use1["Logs, metrics, CNI, or security"]
agent2 --> use2["Logs, metrics, CNI, or security"]
agent3 --> use3["Logs, metrics, CNI, or security"]
Examples: CNI agent, log collector, Node Exporter, runtime security agent, or storage node plugin.
4.10 Job and CronJob
Jobs run tasks to completion. CronJobs create Jobs on a schedule. They are not suitable for long-running services.
flowchart TD
cron["CronJob\nschedule"] --> job1["Job 2026-06-13 01:00"]
cron --> job2["Job 2026-06-14 01:00"]
job1 --> pod1["Pod attempt 1"]
job1 --> pod2["Pod retry, if needed"]
pod1 --> success["Succeeded"]
pod2 --> fail["Failed or Succeeded"]
job2 --> pod3["Pod next run"]
Checkpoints: backoffLimit, activeDeadlineSeconds, concurrencyPolicy, startingDeadlineSeconds, and retention of old Jobs.
4.11 HPA scaling of a workload
The Horizontal Pod Autoscaler observes metrics and adjusts the desired replica count. It does not replace meaningful requests or load tests.
flowchart LR
metrics["Metrics API\nCPU, Memory, or Custom Metrics"] --> hpa["Horizontal Pod Autoscaler"]
hpa --> decision{Need to scale?}
decision -->|more load| up["increase replicas"]
decision -->|less load| down["decrease replicas"]
up --> deploy["Deployment or StatefulSet scale subresource"]
down --> deploy
deploy --> pods["more or fewer Pods"]
Operational relevance: Without Metrics Server or a suitable custom metrics adapter, HPA cannot work meaningfully.
5. Services, DNS, Ingress, and Gateway API
5.1 Pod network: Each Pod has its own IP
The CNI plugin provides Pod networking and routing. Kubernetes defines the model; concrete enforcement depends on the CNI.
flowchart LR
subgraph nodeA["Node A"]
podA1["Pod A1\n10.244.1.10"]
podA2["Pod A2\n10.244.1.11"]
cniA["CNI"]
end
subgraph nodeB["Node B"]
podB1["Pod B1\n10.244.2.10"]
podB2["Pod B2\n10.244.2.11"]
cniB["CNI"]
end
podA1 --> cniA
cniA <--> cniB
cniB --> podB1
podA2 --> cniA
cniB --> podB2
Rule of thumb: Pod IP addresses are routable, but not stable. For stable reachability, use Services.
5.2 Service, selector, and EndpointSlices
A Service finds Pods through labels and provides stable service discovery. EndpointSlices contain the actual routable backends.
flowchart TD
svc["Service\nselector app=shop"] --> selector["Label Selector"]
selector --> pod1["Pod 1\napp=shop\nReady"]
selector --> pod2["Pod 2\napp=shop\nReady"]
selector --> pod3["Pod 3\napp=shop\nNotReady"]
pod1 --> eps["EndpointSlice"]
pod2 --> eps
pod3 -. not ready .-> eps
eps --> dataplane["kube-proxy or eBPF dataplane"]
dataplane --> client["Internal cluster client"]
Inspection question: If a Service does not work, do not inspect only the Service. Inspect labels, selectors, Pod readiness, and EndpointSlices.
5.3 Service types compared
Service types differ in how broadly they expose a service.
flowchart TD
svc["Service"] --> clusterip["ClusterIP\ninternal in the cluster"]
svc --> nodeport["NodePort\nport on every Node"]
svc --> lb["LoadBalancer\nexternal load balancer"]
svc --> external["ExternalName\nDNS alias"]
clusterip --> internal["Pod-to-Service traffic"]
nodeport --> node["Client reaches Node IP and port"]
lb --> cloud["Cloud or MetalLB provides external address"]
external --> dns["CNAME-like name resolution"]
Design rule: Use ClusterIP as the default. Use LoadBalancer, Gateway, or Ingress for controlled external access. NodePort is often an infrastructure detail, not your primary publication model.
5.4 DNS resolution inside the cluster
CoreDNS provides DNS-based service discovery. DNS problems often surface as application errors even when Service and Pods look correct.
sequenceDiagram
participant APP as App Pod
participant DNS as CoreDNS
participant API as API Server
participant SVC as Service
participant EP as EndpointSlice
APP->>DNS: resolve shop.demo.svc.cluster.local
DNS->>API: Service and Endpoint data from cache/API
API-->>DNS: Service IP or Endpoint information
DNS-->>APP: DNS answer
APP->>SVC: connect to Service IP
SVC->>EP: route to matching Pod
Troubleshooting note: Test DNS from a debug Pod in the same Namespace and test both the short name and the FQDN.
5.5 Ingress path for HTTP and HTTPS
Ingress is an API object, but it works only with a running Ingress Controller. The controller translates rules into concrete proxy configuration.
flowchart LR
client["Browser or API Client"] --> dns["DNS"]
dns --> lb["Load Balancer"]
lb --> controller["Ingress Controller"]
controller --> ingress["Ingress rules\nHost and Path"]
ingress --> service["Service"]
service --> endpoints["EndpointSlices"]
endpoints --> pod1["Pod A"]
endpoints --> pod2["Pod B"]
cert["TLS Secret or cert-manager"] --> controller
Check chain: DNS → Load Balancer → Ingress Controller → IngressClass → Ingress rule → Service → EndpointSlice → Pod readiness.
5.6 Gateway API: role-oriented routing
Gateway API separates infrastructure responsibility and application routing more clearly than classic Ingress models.
flowchart TD
infra["Platform team"] --> gatewayClass["GatewayClass"]
infra --> gateway["Gateway\nListener, address, TLS"]
appteam["Application team"] --> route["HTTPRoute, GRPCRoute, or TCPRoute"]
route --> service["Service"]
service --> pods["Pods"]
gatewayClass --> controller["Gateway Controller"]
gateway --> controller
route --> controller
controller --> dataplane["Proxy or Load Balancer"]
dataplane --> service
Rule of thumb: Gateway API is especially useful when platform teams and application teams have different responsibilities.
5.7 NetworkPolicy: default deny and allowed flows
NetworkPolicies allow or deny Pod traffic. They work only if your network plugin enforces NetworkPolicy.
flowchart TD
ns["Namespace demo"] --> defaultDeny["Default Deny Policy"]
defaultDeny --> pod["Pod app=api"]
client["Pod app=frontend"] -->|allowed TCP 8080| pod
db["Pod app=db"] -. not allowed .-> pod
internet["external traffic"] -. not allowed .-> pod
policy["NetworkPolicy\nPodSelector app=api"] --> ingress["Ingress rules"]
ingress --> client
Anti-pattern: A NetworkPolicy without a supporting CNI gives you a false sense of security. Test enforcement in practice.
5.8 Egress control
Egress rules limit where a Pod may establish connections. This matters for data exfiltration, supply-chain protection, and controlled dependencies.
flowchart LR
pod["App Pod"] --> policy["Egress NetworkPolicy"]
policy -->|allowed| dns["CoreDNS TCP/UDP 53"]
policy -->|allowed| api["internal API Service"]
policy -. blocked .-> internet["arbitrary Internet"]
policy -. blocked .-> metadata["Cloud Metadata Service"]
Workshop exercise: Create a default-deny egress policy and open only DNS and the target service. Observe the side effects.
5.9 Service Mesh in context
A Service Mesh adds traffic management, mTLS, telemetry, and policies to Kubernetes. It does not replace Services, Deployments, or RBAC.
flowchart TD
client["Client Pod"] --> sidecarA["Sidecar Proxy A"]
sidecarA --> mesh["Mesh Control Plane"]
sidecarA --> sidecarB["Sidecar Proxy B"]
sidecarB --> server["Server Pod"]
mesh --> certs["Certificates and identities"]
mesh --> routing["Traffic rules"]
mesh --> telemetry["Metrics and traces"]
k8s["Kubernetes API"] --> mesh
Positioning: A mesh increases platform capabilities, but also complexity. Use it when you can name and operate the specific requirements.
6. Configuration, Secrets, and Images
6.1 ConfigMaps: separate configuration from the image
ConfigMaps decouple configuration from the container image. You can use them as environment variables or as files mounted through volumes.
flowchart TD
cm["ConfigMap"] --> env["Environment Variables"]
cm --> volume["Volume Mount as file"]
deploy["Deployment"] --> pod["Pod"]
pod --> env
pod --> volume
image["Container Image"] --> pod
change["ConfigMap changed"] --> effect{How is it mounted?}
effect -->|Env| restart["Pod restart required"]
effect -->|Volume| refresh["file can be updated"]
Rule of thumb: A configuration change does not automatically mean an application reload. Plan the reload mechanism or rollout deliberately.
6.2 Secrets: access path and protection boundaries
Kubernetes Secrets are API objects. Base64 is only encoding, not encryption. Protection depends on RBAC, encryption at rest, Node protection, and secret management.
flowchart TD
user["authorized identity or controller"] --> api["API Server"]
api --> rbac["RBAC check"]
rbac --> secret["Secret object"]
secret --> etcd["etcd\noptional Encryption at Rest"]
secret --> kubelet["kubelet on target node"]
kubelet --> pod["Pod Volume or Env"]
pod --> app["App reads Secret"]
attacker["overbroad permissions"] -. get/list/watch secrets .-> secret
Operational rule: Anyone who can read Secrets often has effective access to target systems. Grant Secret permissions much more restrictively than general read permissions.
6.3 External secret systems
Many platforms synchronize secrets from Vault, cloud secret managers, or SOPS-encrypted Git repositories. Kubernetes remains the execution platform, not necessarily the authoritative source.
flowchart LR
source["External secret source\nVault, Cloud Secret Manager, or SOPS"] --> controller["External Secrets Controller or GitOps"]
controller --> api["Kubernetes API"]
api --> k8ssecret["Kubernetes Secret"]
k8ssecret --> pod["Pod"]
rotate["Rotation"] --> source
source --> audit["Audit and access control"]
Decision: Define where the secret is authoritative, how rotation happens, and who has access to the source, the Kubernetes Secret, and the Pod.
6.4 Image pull and registry access
ImagePullBackOff is often caused by incorrect image names, tags, registry permissions, network problems, or missing pull Secrets.
sequenceDiagram
participant K as kubelet
participant R as Container Runtime
participant REG as Registry
participant P as Pod
K->>R: Pull image
R->>REG: Request manifest and layers
REG-->>R: allowed or denied
R-->>K: image available or error
K->>P: Start container
P-->>K: Report status
Checkpoints: Image name, tag, architecture, registry reachability, TLS, authentication, imagePullSecrets, quotas, and rate limits.
6.5 Image supply chain
A production-ready image is not created at deployment time. It needs build control, scans, signatures, and traceable artifacts.
flowchart LR
code["Source code"] --> build["Build Pipeline"]
build --> test["Tests"]
test --> scan["Image and dependency scan"]
scan --> sbom["SBOM"]
sbom --> sign["Signature"]
sign --> registry["Registry"]
registry --> admission["Admission Policy"]
admission --> cluster["Cluster Deployment"]
Review question: Which policy prevents untested or unsigned images from entering production Namespaces?
7. Storage and Data Persistence
7.1 Basic storage model
Pods are ephemeral. Persistence is provided through volumes, PVCs, PVs, StorageClasses, and CSI drivers.
flowchart TD
pod["Pod"] --> volume["Volume Mount"]
volume --> pvc["PersistentVolumeClaim"]
pvc --> pv["PersistentVolume"]
pv --> backend["Storage Backend"]
sc["StorageClass"] --> pvc
csi["CSI Driver"] --> pv
csi --> backend
Rule of thumb: The application requests storage through a PVC. The platform decides StorageClass, provisioning, performance, backup, and lifecycle.
7.2 Dynamic volume provisioning
With dynamic provisioning, a CSI provisioner creates the appropriate backend volume when a PVC is created.
sequenceDiagram
participant U as You
participant API as API Server
participant PVC as PVC Controller
participant CSI as CSI Provisioner
participant BE as Storage Backend
participant POD as Pod
U->>API: Create PVC
API-->>PVC: new Claim
PVC->>CSI: request Volume
CSI->>BE: create Storage Volume
BE-->>CSI: Volume provisioned
CSI->>API: create and bind PV
POD->>API: start Pod with PVC
API-->>POD: PVC is bound
Troubleshooting note: For Pending PVCs, inspect StorageClass, provisioner, events, quotas, zones, and CSI controller logs.
7.3 Binding modes and zones
Storage can be bound immediately or only at scheduling time. This matters when storage is tied to availability zones.
flowchart TD
pvc["PVC"] --> mode{volumeBindingMode}
mode --> immediate["Immediate"]
mode --> wait["WaitForFirstConsumer"]
immediate --> pv1["PV is created immediately"]
pv1 --> risk["possible zone does not match later Node"]
wait --> pod["Pod is created"]
pod --> scheduler["Scheduler selects Node and zone"]
scheduler --> pv2["PV matched to the zone"]
Design rule: For zonal cloud storage, WaitForFirstConsumer is often the more robust choice.
7.4 StatefulSet with volumeClaimTemplates
StatefulSets create a dedicated Claim per Pod. When a Pod is deleted, the volume usually remains.
flowchart TD
sts["StatefulSet postgres"] --> template["volumeClaimTemplates"]
template --> pvc0["data-postgres-0"]
template --> pvc1["data-postgres-1"]
template --> pvc2["data-postgres-2"]
pvc0 --> pv0["PV 0"]
pvc1 --> pv1["PV 1"]
pvc2 --> pv2["PV 2"]
pod0["postgres-0"] --> pvc0
pod1["postgres-1"] --> pvc1
pod2["postgres-2"] --> pvc2
Inspection question: What happens to PVCs when you delete the StatefulSet? Do not rely on assumptions. Check the retention policy and platform behavior.
7.5 CSI components
CSI separates Kubernetes from specific storage backends. Controller components manage volumes; Node components mount them on Nodes.
flowchart TD
subgraph control["CSI Controller layer"]
provisioner["external-provisioner"]
attacher["external-attacher"]
resizer["external-resizer"]
snapshotter["external-snapshotter"]
csiController["CSI Controller Plugin"]
end
subgraph node["CSI Node layer"]
kubelet["kubelet"]
registrar["node-driver-registrar"]
csiNode["CSI Node Plugin"]
mount["Mount into Pod"]
end
provisioner --> csiController
attacher --> csiController
resizer --> csiController
snapshotter --> csiController
csiController --> backend["Storage Backend"]
kubelet --> registrar
kubelet --> csiNode
csiNode --> mount
mount --> pod["Pod"]
Operational relevance: Storage problems can originate in the PVC, PV, StorageClass, CSI Controller, CSI Node Plugin, backend, or Node mount.
7.6 Backup and restore for workloads
Kubernetes manifests alone are not enough for stateful applications. You also need data backups and restore tests.
flowchart LR
manifests["Manifests and Helm Values"] --> git["Git or artifact repository"]
pvdata["Persistent data"] --> backup["Backup or Snapshot"]
etcd["Cluster state"] --> etcdbackup["etcd Backup"]
git --> restore["Restore process"]
backup --> restore
etcdbackup --> restore
restore --> validation["Validation\ndata, app, Service, permissions"]
Rule of thumb: A backup is dependable only after you have practiced the restore and validated it from the application perspective.
8. Scheduling, Resources, and Availability
8.1 Scheduling flow
The Scheduler selects a Node based on resources, constraints, policies, and current cluster conditions.
sequenceDiagram
participant API as API Server
participant S as Scheduler
participant N as Nodes
participant K as kubelet
API-->>S: Pod without nodeName is visible
S->>N: Filter Nodes
S->>N: Score Nodes
S->>API: Bind Pod to Node
API-->>K: kubelet on target Node sees Pod
K->>K: Prepare and start containers
K->>API: Report Pod status
Inspection question: A Pod in Pending often has not even reached kubelet yet. Start with Scheduler events.
8.2 Requests, Limits, and QoS
Requests drive scheduling. Limits cap runtime consumption. Together they determine QoS classes that matter during resource pressure.
flowchart TD
container["Container resources"] --> requests["Requests"]
container --> limits["Limits"]
requests --> scheduling["Scheduler plans capacity"]
limits --> runtime["Runtime limits consumption"]
requests --> qos{QoS class}
limits --> qos
qos --> guaranteed["Guaranteed"]
qos --> burstable["Burstable"]
qos --> besteffort["BestEffort"]
besteffort --> eviction["higher eviction risk"]
Design rule: Every production workload needs meaningful Requests. Without Requests, Kubernetes plans capacity blindly.
8.3 Node Affinity and Pod Anti-Affinity
Affinity rules control placement. Use hard rules sparingly; otherwise you create unnecessary Pending problems.
flowchart TD
pod["Pod"] --> required["requiredDuringScheduling"]
pod --> preferred["preferredDuringScheduling"]
required --> nodeA["only Nodes with label disk=ssd"]
preferred --> nodeB["prefer zone=eu-central-1a"]
anti["Pod Anti-Affinity"] --> spread["not next to same app Pod"]
spread --> nodes["spread across Nodes or zones"]
Workshop note: Have participants set a hard Affinity that no Node satisfies. They should then interpret the Events.
8.4 Taints and Tolerations
Taints repel Pods. Tolerations allow Pods to run on tainted Nodes, but do not guarantee placement there.
flowchart LR
node["Node with Taint\ndedicated=gpu:NoSchedule"] --> barrier["rejection"]
podA["Pod without Toleration"] -. not scheduled .-> barrier
podB["Pod with matching Toleration"] --> barrier
barrier --> scheduler["Scheduler may consider Node"]
scheduler --> node
Rule of thumb: Toleration means “may run there,” not “must run there.” For targeted placement, combine Taints/Tolerations with Node Affinity.
8.5 Topology Spread Constraints
Topology Spread Constraints spread Pods across failure domains such as Nodes, zones, or racks.
flowchart TD
deploy["Deployment replicas=6"] --> tsc["Topology Spread Constraint\ntopologyKey=zone"]
tsc --> zoneA["Zone A\n2 Pods"]
tsc --> zoneB["Zone B\n2 Pods"]
tsc --> zoneC["Zone C\n2 Pods"]
zoneA --> nodeA1["Node A1"]
zoneB --> nodeB1["Node B1"]
zoneC --> nodeC1["Node C1"]
Design question: Should Kubernetes still schedule when the topology is uneven, or should it block hard? You control that with whenUnsatisfiable.
8.6 PodDisruptionBudget
A PDB protects against voluntary disruptions such as drain, upgrade, or manual evictions. It does not protect against Node failure or crashes.
flowchart LR
deploy["Deployment replicas=3"] --> pdb["PDB minAvailable=2"]
drain["kubectl drain node-a"] --> eviction1["Eviction Pod 1"]
eviction1 --> allowed{Will at least 2 remain available?}
allowed -->|yes| evict["Eviction allowed"]
allowed -->|no| block["Eviction blocked"]
pdb --> allowed
Operational relevance: Overly strict PDBs can block upgrades. Weak PDBs do not adequately protect the application.
8.7 Cluster Autoscaler and Pending Pods
The Cluster Autoscaler reacts to Pods that cannot be scheduled because of missing Node capacity. It does not fix incorrect Affinity, Taints, or PVC issues.
flowchart TD
pod["Pod Pending"] --> reason{Why Pending?}
reason -->|not enough CPU or Memory| ca["Cluster Autoscaler checks Node group"]
reason -->|unsatisfiable Affinity| fix1["correct manifest or labels"]
reason -->|missing Toleration| fix2["check Toleration or Taint design"]
reason -->|storage zone conflict| fix3["check StorageClass and binding"]
ca --> scale["add new Node"]
scale --> scheduler["Scheduler places Pod"]
Rule of thumb: Autoscaling does not replace clean scheduling design.
8.8 ResourceQuotas and LimitRanges
Namespaces can be constrained by Quotas and LimitRanges. This protects platform capacity and enforces minimum standards.
flowchart TD
ns["Namespace team-a"] --> quota["ResourceQuota"]
ns --> limitrange["LimitRange"]
quota --> max["maximum total\nCPU, Memory, Pods, PVCs"]
limitrange --> defaults["Default Requests and Limits"]
deploy["new Workload"] --> admission["Admission check"]
max --> admission
defaults --> admission
admission -->|ok| create["Resource is created"]
admission -->|violates limit| reject["Request is rejected"]
Review question: Does each team Namespace have Quotas, default Requests, and clear escalation paths for more capacity?
9. Security and Access Control
9.1 Security control chain
Security in Kubernetes is a chain. A single mechanism is insufficient.
flowchart LR
request["API Request"] --> authn["Authentication"]
authn --> authz["Authorization\nRBAC or Webhook"]
authz --> admission["Admission\nvalidating and mutating"]
admission --> etcd["Persistence in etcd"]
etcd --> kubelet["Execution by kubelet"]
kubelet --> runtime["Runtime isolation"]
runtime --> network["NetworkPolicy"]
network --> observe["Audit, Logs, Runtime Detection"]
Rule of thumb: RBAC answers “may this identity perform this API action?” Pod Security answers “is this Pod specification allowed?” NetworkPolicy answers “which Pod traffic is allowed?”
9.2 RBAC relationship model
RBAC connects subjects with roles. Roles contain rules. Bindings establish the connection.
flowchart TD
subject["Subject\nUser, Group, or ServiceAccount"] --> binding["RoleBinding or ClusterRoleBinding"]
binding --> role["Role or ClusterRole"]
role --> rule1["apiGroups"]
role --> rule2["resources"]
role --> rule3["verbs"]
rule3 --> examples["get, list, watch, create, update, patch, delete"]
binding --> scope{Scope}
scope --> ns["Namespace"]
scope --> cluster["Cluster-wide"]
Anti-pattern: cluster-admin for CI/CD, developers, or workloads is almost always too broad.
9.3 ServiceAccount and workload identity
Pods use ServiceAccounts to talk to the Kubernetes API. Permissions are attached through RBAC bindings, not through the Pod name.
flowchart LR
pod["Pod"] --> sa["ServiceAccount"]
sa --> token["Bound ServiceAccount Token"]
token --> api["API Server"]
api --> rbac["RBAC"]
rbac --> allowed["allowed API actions"]
rbac --> denied["denied API actions"]
Check command: kubectl auth can-i --as system:serviceaccount:<namespace>:<serviceaccount> <verb> <resource>.
9.4 Admission Control and policy enforcement
Admission Controllers check or modify objects before they are stored. This lets you enforce platform standards.
sequenceDiagram
participant U as You or CI/CD
participant API as API Server
participant M as Mutating Admission
participant V as Validating Admission
participant P as Policy Engine
participant E as etcd
U->>API: Create Pod
API->>M: add defaults or sidecars
M-->>API: mutated object
API->>V: validate object
V->>P: check policy
P-->>V: allowed or denied
V-->>API: result
API->>E: store if allowed
Examples: Pod Security Admission, ValidatingAdmissionPolicy, Kyverno, Gatekeeper, image policies, Namespace standards.
9.5 Pod Security Standards
Pod Security Standards structure Pod security requirements into levels. Enforcement is often done through Namespace labels and Pod Security Admission.
flowchart TD
ns["Namespace"] --> labels["pod-security.kubernetes.io labels"]
labels --> privileged["Privileged\nvery open"]
labels --> baseline["Baseline\nbasic restrictions"]
labels --> restricted["Restricted\nmore strongly hardened"]
pod["new Pod"] --> admission["Pod Security Admission"]
labels --> admission
admission -->|compliant| allowed["Pod is admitted"]
admission -->|violates policy| denied["Pod is rejected or warned"]
Workshop note: Start with warn and audit before enforcing restricted in production Namespaces.
9.6 SecurityContext in the Pod
The SecurityContext controls Linux and runtime security options at Pod or container level.
flowchart TD
pod["Pod spec"] --> podctx["Pod SecurityContext"]
pod --> containers["Containers"]
containers --> cctx["Container SecurityContext"]
podctx --> fs["fsGroup, runAsUser, runAsNonRoot"]
cctx --> caps["Capabilities drop/add"]
cctx --> priv["privileged"]
cctx --> ro["readOnlyRootFilesystem"]
cctx --> seccomp["seccompProfile"]
cctx --> allow["allowPrivilegeEscalation"]
Review rule: By default: non-root, no privilege escalation, no unnecessary capabilities, preferably read-only root filesystem, and an appropriate seccomp profile.
9.7 Tenant isolation in the Namespace model
Namespaces alone are not a complete security boundary. Only RBAC, Quotas, NetworkPolicies, Pod Security, Admission, and separate ownership turn them into a useful tenancy model.
flowchart TD
cluster["Cluster"] --> nsA["Namespace team-a"]
cluster --> nsB["Namespace team-b"]
nsA --> rbacA["RBAC team-a"]
nsA --> quotaA["Quota and LimitRange"]
nsA --> netA["NetworkPolicies"]
nsA --> pssA["Pod Security"]
nsB --> rbacB["RBAC team-b"]
nsB --> quotaB["Quota and LimitRange"]
nsB --> netB["NetworkPolicies"]
nsB --> pssB["Pod Security"]
admission["Cluster-wide Admission Policies"] --> nsA
admission --> nsB
Decision: For very different security or compliance requirements, a separate cluster may be more appropriate than a shared multi-tenant cluster.
9.8 Audit logs and runtime detection
Audit logs show API activity. Runtime detection shows behavior inside running containers and on Nodes.
flowchart LR
actor["User, CI/CD, or Workload"] --> api["API Server"]
api --> audit["Audit Log"]
pod["running Pod"] --> runtime["Runtime Events"]
node["Node"] --> syscall["System calls and process behavior"]
runtime --> detection["Falco, Tetragon, or EDR"]
syscall --> detection
audit --> siem["SIEM or log platform"]
detection --> siem
siem --> alert["Security Alert"]
Rule of thumb: API audit answers who changed Kubernetes. Runtime detection answers what happens inside the running container or Node.
10. Observability and Operations
10.1 Three signal types: metrics, logs, and traces
Observability combines multiple signal types. Events and Kubernetes status complement this perspective.
flowchart TD
workload["Workload"] --> metrics["Metrics"]
workload --> logs["Logs"]
workload --> traces["Traces"]
k8s["Kubernetes API"] --> status["Status and Conditions"]
k8s --> events["Events"]
metrics --> dash["Dashboards and Alerts"]
logs --> search["Log search"]
traces --> flow["Request flows"]
status --> triage["Triage"]
events --> triage
Operational rule: Start triage most often with status and events, not with a blind log dump.
10.2 Prometheus model inside the cluster
Prometheus collects metrics from applications, Nodes, and Kubernetes components. Service discovery typically uses Kubernetes labels and Endpoints.
flowchart TD
prom["Prometheus"] --> sd["Kubernetes Service Discovery"]
sd --> pods["App Pods with /metrics"]
sd --> nodes["Node Exporter"]
sd --> kubelet["kubelet / cAdvisor"]
sd --> ksm["kube-state-metrics"]
prom --> rules["Alert Rules"]
rules --> alertmanager["Alertmanager"]
prom --> grafana["Grafana Dashboards"]
Review question: Do you measure only infrastructure state, or also business-relevant SLO signals such as error rate, latency, and throughput?
10.3 Logging pipeline
Containers usually write logs to stdout and stderr. Node agents collect those logs and forward them to a central platform.
flowchart LR
app["App Container"] --> stdout["stdout and stderr"]
stdout --> nodefiles["Node log files"]
nodefiles --> agent["Log Agent as DaemonSet"]
agent --> buffer["Buffer and parsing"]
buffer --> backend["Loki, Elasticsearch, or Cloud Logging"]
backend --> query["Search and dashboards"]
Anti-pattern: Writing logs only to the container filesystem is not robust. Pod restarts or Node problems can make you lose critical diagnostic information.
10.4 Tracing across services
Tracing helps make request paths across multiple Services visible.
sequenceDiagram
participant C as Client
participant FE as Frontend
participant API as Backend API
participant DB as Database
participant OTEL as OpenTelemetry Collector
C->>FE: Request with Trace Context
FE->>OTEL: Send span
FE->>API: Forward request
API->>OTEL: Send span
API->>DB: Query
DB-->>API: Result
API-->>FE: Response
FE-->>C: Response
OTEL->>OTEL: Assemble trace
Design rule: Traces are useful only if applications propagate context and emit relevant spans.
10.5 Events as a triage entry point
Events are time-ordered hints from Kubernetes components. They often explain why scheduling, image pull, volumes, or probes fail.
flowchart TD
problem["Problem visible"] --> events["kubectl get events --sort-by=.lastTimestamp"]
events --> scheduling["FailedScheduling"]
events --> image["FailedPull or BackOff"]
events --> volume["FailedMount or FailedAttachVolume"]
events --> probe["Unhealthy Probe"]
events --> eviction["Evicted or NodePressure"]
scheduling --> next1["check resources and constraints"]
image --> next2["check registry and Pull Secret"]
volume --> next3["check PVC, PV, CSI"]
probe --> next4["check app health and Probe config"]
eviction --> next5["check Node resources"]
Rule of thumb: Events are not durable logging. Collect them centrally if you need longer lookbacks.
11. Platform Lifecycle, GitOps, and Add-ons
11.1 kubeadm bootstrap flow
kubeadm initializes the Control Plane and allows additional Nodes to join. It does not automatically provision your machines, network environment, or platform add-ons.
sequenceDiagram
participant A as Admin
participant K as kubeadm
participant CP as Control Plane Node
participant API as API Server
participant W as Worker Node
A->>K: kubeadm init
K->>CP: generate certificates and configuration
K->>CP: write static Pods for Control Plane
CP->>API: API Server becomes available
A->>K: install CNI
A->>W: kubeadm join with token
W->>API: register Node
API-->>A: Cluster shows Node Ready after CNI and kubelet
Inspection question: Why is a cluster not fully usable after kubeadm init? Because networking, add-ons, storage, ingress, policies, and observability can still be missing.
11.2 Cluster upgrade with drain
A controlled upgrade respects PDBs, DaemonSets, stateful workloads, and rollback strategies.
flowchart TD
plan["Plan upgrade"] --> backup["check etcd backup and restore path"]
backup --> cp["upgrade Control Plane"]
cp --> node1["Node cordon"]
node1 --> drain["Node drain"]
drain --> upgrade["upgrade kubelet and runtime"]
upgrade --> uncordon["Node uncordon"]
uncordon --> validate["check workloads and SLOs"]
validate --> next["next Node"]
Operational rule: An upgrade is not just a package change. It is a planned platform process with validation, a communication window, and a fallback path.
11.3 GitOps flow
GitOps makes Git the authoritative source for declarative configuration. A controller synchronizes the cluster with the repository.
flowchart LR
dev["You change manifest or values"] --> pr["Pull Request"]
pr --> review["Review and checks"]
review --> merge["Merge into target branch"]
merge --> repo["Git Repository"]
repo --> gitops["Argo CD or Flux"]
gitops --> diff["Diff against cluster"]
diff --> sync["Synchronization"]
sync --> api["Kubernetes API"]
api --> status["Health and Sync Status"]
Rule of thumb: GitOps does not replace architecture decisions. It enforces traceability, review, and reproducible changes.
11.4 Helm release lifecycle
Helm renders templates into Kubernetes manifests and manages releases. Failures can occur in the chart, values, rendered output, or Kubernetes apply phase.
flowchart TD
chart["Helm Chart"] --> render["helm template"]
values["values.yaml"] --> render
render --> manifests["rendered Manifests"]
manifests --> lint["lint and policy checks"]
lint --> upgrade["helm upgrade --install"]
upgrade --> api["Kubernetes API"]
api --> release["Helm Release Secret"]
api --> resources["Kubernetes resources"]
Check chain: Chart version → Values → rendered YAML → server-side validation → rollout status → app health.
11.5 Kustomize overlay model
Kustomize modifies base manifests through overlays. This is useful for environments without introducing a templating language.
flowchart TD
base["base\nDeployment, Service, Config"] --> dev["overlay dev"]
base --> stage["overlay stage"]
base --> prod["overlay prod"]
dev --> patchDev["Patches and images dev"]
stage --> patchStage["Patches and images stage"]
prod --> patchProd["Patches and images prod"]
patchProd --> output["kubectl apply -k overlays/prod"]
Design rule: Keep the base stable and overlays small. Large, contradictory patches make deployments hard to review.
11.6 Add-on dependencies
Cluster add-ons are themselves workloads and can depend on each other. Failure in one add-on often produces apparently unrelated workload failures.
flowchart TD
cni["CNI"] --> pods["Pods receive networking"]
coredns["CoreDNS"] --> discovery["Service Discovery"]
metrics["Metrics Server"] --> hpa["HPA and kubectl top"]
ingress["Ingress or Gateway Controller"] --> external["external traffic"]
csi["CSI"] --> volumes["Persistent Volumes"]
cert["cert-manager"] --> tls["TLS certificates"]
externaldns["ExternalDNS"] --> dns["DNS Records"]
observability["Monitoring and Logging"] --> ops["Operations and Triage"]
Rule of thumb: Platform add-ons need the same lifecycle as applications: versioning, monitoring, upgrade plan, and rollback.
12. Troubleshooting Diagrams
12.1 General Kubernetes triage flow
Start broad, then narrow down systematically. Do not jump straight into container logs if the Pod is not yet running.
flowchart TD
start["Problem reported"] --> scope["Clarify scope\nNamespace, Workload, time"]
scope --> get["kubectl get relevant resources"]
get --> describe["kubectl describe"]
describe --> events["Sort Events"]
events --> status{Resource status?}
status -->|Pending| pending["Scheduling, resources, PVC, Taints"]
status -->|CrashLoop| crash["Logs previous, Exit Code, Probes"]
status -->|Running but unreachable| network["Service, EndpointSlice, DNS, Ingress"]
status -->|Forbidden| rbac["RBAC and ServiceAccount"]
status -->|Node affected| node["Node, kubelet, Runtime, CNI"]
pending --> fix["Correction and validation"]
crash --> fix
network --> fix
rbac --> fix
node --> fix
Rule: Document cause, change, and validation. Fixing symptoms alone is not operationally mature.
12.2 Pod Pending
Pending means the Pod is not successfully running on a Node. The cause can be before or during Node assignment.
flowchart TD
pending["Pod Pending"] --> events["Check Events"]
events --> scheduling{FailedScheduling?}
scheduling -->|yes| resources["CPU, Memory, Pod Quota"]
scheduling -->|yes| constraints["Affinity, Taints, Topology"]
scheduling -->|no| image["Image pull not completed yet?"]
events --> volume{Volume Events?}
volume -->|yes| pvc["PVC, PV, StorageClass, CSI"]
resources --> fix["Correct Requests or capacity"]
constraints --> fix
pvc --> fix
image --> fix
Command entry: kubectl describe pod <pod> -n <namespace> and kubectl get events -n <namespace> --sort-by=.lastTimestamp.
12.3 CrashLoopBackOff
CrashLoopBackOff occurs when a container repeatedly starts and exits.
flowchart TD
crash["CrashLoopBackOff"] --> previous["kubectl logs --previous"]
previous --> exit["Check Exit Code"]
exit --> config["Check ConfigMap, Secret, Env"]
exit --> deps["Check dependencies\nDB, API, DNS"]
exit --> probe["Check Probe configuration"]
exit --> resources["Check OOMKilled or CPU starvation"]
config --> fix["Deploy fix"]
deps --> fix
probe --> fix
resources --> fix
fix --> rollout["Monitor rollout"]
Anti-pattern: “Fixing” the restart backoff only by deleting the Pod does not address the root cause.
12.4 ImagePullBackOff
ImagePullBackOff indicates that kubelet or the runtime cannot pull the image.
flowchart TD
img["ImagePullBackOff"] --> name["Check image name and tag"]
name --> registry["Registry reachable?"]
registry --> auth["Pull Secret correct?"]
auth --> tls["TLS and certificates ok?"]
tls --> arch["Architecture matches? amd64, arm64"]
arch --> quota["Rate limit or Registry quota?"]
quota --> fix["Correct manifest, Secret, or Registry"]
Check command: kubectl describe pod usually shows the exact registry error message in the Event section.
12.5 Service without Endpoints
A Service without Endpoints is often a label, selector, or readiness problem.
flowchart TD
svc["Service does not respond"] --> eps["Check EndpointSlices"]
eps --> empty{No Endpoints?}
empty -->|yes| selector["Check Service selector"]
selector --> labels["Check Pod labels"]
labels --> readiness["Check Pod readiness"]
readiness --> ports["Check targetPort and containerPort"]
empty -->|no| dataplane["Check kube-proxy, eBPF, or NetworkPolicy"]
ports --> fix["Correct labels, readiness, or ports"]
dataplane --> fix
Command entry: kubectl get svc,endpoints,endpointslices,pods -n <namespace> -o wide.
12.6 DNS errors in the cluster
DNS errors can come from incorrect names, CoreDNS problems, NetworkPolicies, or Node network issues.
flowchart TD
dnsfail["DNS resolution fails"] --> debugpod["Start debug Pod in Namespace"]
debugpod --> fqdn["Test short name and FQDN"]
fqdn --> coredns["Check CoreDNS Pods and logs"]
coredns --> svc["Check kube-dns Service"]
svc --> netpol["Does NetworkPolicy allow DNS?"]
netpol --> node["Check Node network and CNI"]
node --> fix["Correct cause and retest"]
Inspection question: Does the problem affect all Namespaces or only one Namespace? That answer strongly narrows down CoreDNS versus policy issues.
12.7 Ingress or Gateway does not route
External HTTP(S) traffic has many handoff points. Check them in order.
flowchart TD
fail["External request fails"] --> dns["DNS record correct?"]
dns --> lb["Load Balancer reachable?"]
lb --> controller["Ingress or Gateway Controller running?"]
controller --> routeClass["IngressClass or GatewayClass correct?"]
routeClass --> rule["Host, Path, Listener, Route correct?"]
rule --> svc["Service correct?"]
svc --> eps["EndpointSlices present?"]
eps --> pod["Pods ready and app responds?"]
Operational rule: For each public service, document DNS name, TLS source, controller, routing object, Service, and ownership.
12.8 PVC Pending
PVCs remain Pending when binding, provisioning, or the storage backend does not fit.
flowchart TD
pvc["PVC Pending"] --> sc["StorageClass exists?"]
sc --> provisioner["Provisioner available?"]
provisioner --> quota["StorageQuota or backend capacity?"]
quota --> binding["volumeBindingMode and zone?"]
binding --> access["AccessMode suitable?"]
access --> events["Check PVC and CSI Events"]
events --> logs["Check CSI Controller logs"]
Check command: kubectl describe pvc <name> -n <namespace> is the first entry point.
12.9 Node NotReady
Node problems often affect multiple workloads. kubelet, runtime, networking, disk, memory, and certificates are central checkpoints.
flowchart TD
node["Node NotReady"] --> cond["Check Node Conditions"]
cond --> kubelet["kubelet status and logs"]
cond --> runtime["Container Runtime reachable?"]
cond --> cni["CNI ready?"]
cond --> pressure["DiskPressure, MemoryPressure, PIDPressure"]
cond --> cert["Certificates and API connection"]
kubelet --> fix["Correct service or configuration"]
runtime --> fix
cni --> fix
pressure --> cleanup["free capacity or replace Node"]
cert --> fix
Operational rule: For Node problems, always check whether evictions, PDBs, and rescheduling behave as expected.
12.10 RBAC Forbidden
Forbidden is usually not a network error. It is an authorization decision.
flowchart TD
forbidden["API reports Forbidden"] --> identity["Check identity\nUser, Group, ServiceAccount"]
identity --> verb["Check verb\nget, list, create, patch"]
verb --> resource["Check resource and API Group"]
resource --> scope["Namespace or cluster scope?"]
scope --> binding["Check RoleBinding or ClusterRoleBinding"]
binding --> cani["Test kubectl auth can-i"]
cani --> fix["Grant minimal matching permissions"]
Anti-pattern: Do not reflexively create a ClusterRoleBinding to cluster-admin. Build the smallest appropriate rule.
12.11 OOMKilled
OOMKilled means the container was terminated because of memory limits or Node pressure.
flowchart TD
oom["OOMKilled"] --> limits["Check Memory Limit"]
limits --> usage["Check actual Memory usage"]
usage --> leak["Memory leak or load spike?"]
usage --> request["Request realistic?"]
request --> qos["Check QoS class"]
qos --> node["Check Node MemoryPressure"]
leak --> fix["Correct app or Limit"]
request --> fix
node --> fix
Rule of thumb: A higher memory limit can reduce symptoms, but it does not fix a leak or an incorrect capacity model.
12.12 Stuck rollout
A rollout often gets stuck because new Pods do not become ready or old Pods cannot be removed.
flowchart TD
rollout["Rollout stuck"] --> status["kubectl rollout status"]
status --> rs["Compare ReplicaSets"]
rs --> newpods["Check new Pods"]
newpods --> pending["Pending?"]
newpods --> crash["CrashLoop or Probe Fail?"]
newpods --> readiness["Readiness never becomes true?"]
rs --> pdb["PDB blocks old Pod eviction?"]
pending --> fix["Fix scheduling or storage"]
crash --> fix
readiness --> fix
pdb --> fix
Inspection question: Is the problem in the new ReplicaSet, Service traffic, or the rollout strategy?
12.13 Incident response in a Kubernetes cluster
Incident response needs technical containment and evidence preservation. Do not delete suspicious resources too quickly if you need forensic information.
stateDiagram-v2
[*] --> Detect
Detect --> Scope: identify affected resources
Scope --> Contain: isolate network, permissions, or Node
Contain --> PreserveEvidence: preserve logs, events, images, manifests
PreserveEvidence --> RevokeAccess: rotate tokens, Secrets, roles
RevokeAccess --> Remediate: remove root cause
Remediate --> Recover: controlled restore or rollout
Recover --> Improve: tighten policies and monitoring
Improve --> [*]
Operational rule: Prepare runbooks before you have an incident. Improvisation under time pressure leads to data loss and incomplete root-cause analysis.
13. Workshop Exercises with Diagrams
13.1 Exercise: visually trace a Deployment
The goal is to understand resource relationships in the cluster, not just memorize them.
flowchart LR
task["Roll out Deployment"] --> observe1["kubectl get deploy,rs,pod"]
observe1 --> labels["Check labels and selector"]
labels --> service["Add Service"]
service --> endpoints["Observe EndpointSlices"]
endpoints --> rollout["Change image and follow rollout"]
rollout --> explain["Annotate diagram with real names"]
Acceptance: You can explain which object creates which other object and why a Service continues to work after a rollout.
13.2 Exercise: make NetworkPolicy visible
NetworkPolicies are abstract. This exercise makes allowed and blocked paths visible.
flowchart TD
frontend["frontend Pod"] --> api["api Pod"]
api --> db["db Pod"]
debug["debug Pod"] -. blocked .-> db
internet["Internet"] -. blocked .-> api
policy1["Policy frontend to api"] --> frontend
policy1 --> api
policy2["Policy api to db"] --> api
policy2 --> db
Acceptance: You can use a debug Pod to show which traffic is allowed and which is blocked.
13.3 Exercise: analyze a storage failure
This exercise connects PVC, Events, StorageClass, and CSI components.
flowchart LR
create["Create PVC"] --> pending["Create PVC Pending"]
pending --> inspect["describe pvc and events"]
inspect --> sc["correct StorageClass"]
sc --> bind["PVC Bound"]
bind --> pod["start Pod with mount"]
pod --> verify["write file and recreate Pod"]
Acceptance: You can explain whether the error was in the Claim, StorageClass, provisioner, or backend.
13.4 Exercise: validate the security chain
This exercise shows that security consists of multiple controls.
flowchart TD
sa["ServiceAccount with minimal permissions"] --> cani["kubectl auth can-i"]
cani --> pod["Pod with SecurityContext"]
pod --> admission["Test Pod Security Admission"]
admission --> secret["Check Secret access"]
secret --> netpol["Apply NetworkPolicy"]
netpol --> audit["Review Audit or Event traces"]
Acceptance: You can justify which control limits API access, Pod specification, Secret access, and network traffic.
14. Selecting diagrams by workshop situation
| Situation | Suitable diagrams | Goal |
|---|---|---|
| Introduction to Kubernetes | Big picture, Control Plane and Worker Nodes, Desired State | Build the mental model |
| First Deployments | Deployment, rollout, Pod lifecycle, Service and EndpointSlices | Understand resource relationships |
| Networking lab | Service types, DNS, Ingress, Gateway API, NetworkPolicy | Make traffic paths visible |
| Storage lab | PVC, dynamic provisioning, CSI, StatefulSet | Understand persistence and binding |
| Security lab | API request lifecycle, RBAC, Admission, Pod Security, Secrets | Separate controls clearly |
| Operations and triage | Events, Pending, CrashLoop, Node NotReady, stuck rollout | Narrow failures systematically |
| Platform design | HA Control Plane, GitOps, add-ons, upgrade, tenant isolation | Discuss architecture decisions |
| Incident response | Audit, runtime detection, incident response state machine | Structure response activities |
15. Mermaid conventions for your own additions
15.1 Naming
Use descriptive labels and short technical identifiers. Node IDs should stay ASCII; labels can contain natural language text.
flowchart LR
good["clear label\nService selects Pods by labels"] --> better["easier to explain in workshops"]
bad["svc1"] -. too cryptic .-> unclear["unclear meaning"]
15.2 Do not overload diagrams
A diagram should have one main message. If you need more than three layers of meaning, split the image.
flowchart TD
topic["One topic"] --> diag1["Diagram 1\nArchitecture"]
topic --> diag2["Diagram 2\nFlow"]
topic --> diag3["Diagram 3\nTroubleshooting"]
15.3 Adapt technical depth to the audience
For beginners, show relationships. For advanced learners, add events, status, controllers, and constraints.
flowchart LR
beginner["Fundamentals"] --> simple["Pods, Services, Deployments"]
intermediate["Intermediate"] --> flow["Rollouts, Probes, Storage, Networking"]
professional["Professional"] --> ops["HA, Upgrades, Add-ons, Observability"]
security["Security"] --> controls["RBAC, Admission, Policies, Incident Response"]
16. Sources and technical orientation
This diagram file is based on the official Kubernetes concepts and references. Use these links as technical validation when you deepen the diagrams in a workshop or adapt them to a concrete platform.
- Kubernetes Components
- Cluster Architecture
- Nodes
- Communication between Nodes and the Control Plane
- Services
- EndpointSlices
- Ingress
- Gateway API
- Network Policies
- Persistent Volumes
- Storage Classes
- Dynamic Volume Provisioning
- RBAC
- RBAC Good Practices
- Pod Security Standards
- Pod Security Admission
- Admission Controllers
- Operating etcd clusters for Kubernetes
- Creating Highly Available Clusters with kubeadm