Concepts: Kubernetes
Workshop module for Kubernetes fundamentals, architectural understanding, and operational thinking.
Version: 2026-06-13.
1. Target Picture
This document explains the core Kubernetes concepts in a way that lets you not only operate Kubernetes, but also classify it architecturally. It complements the architecture guide, the best-practices guide, and the command reference. This module focuses on terminology, relationships, responsibility boundaries, and common misconceptions.
After this module, you should be able to do the following:
- explain what Kubernetes provides as a platform and what deliberately remains outside Kubernetes;
- understand the difference between desired state, actual state, and observed status;
- read a manifest and explain the meaning of
apiVersion,kind,metadata,spec, andstatus; - distinguish Pods, Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs from a design perspective;
- classify Services, DNS, Ingress, Gateway API, and NetworkPolicies as networking building blocks;
- understand storage through Volumes, PVCs, PVs, StorageClasses, and CSI;
- recognize why Requests, Limits, scheduling rules, quotas, and autoscaling are required for stable operations;
- explain RBAC, ServiceAccounts, Pod Security Standards, and Admission Control as security layers;
- map common failure symptoms to the affected conceptual area.
The guiding idea is this: Kubernetes is not a single tool. It is an API-centered control system for distributed workloads.
flowchart LR user[You describe the desired state] --> api[Kubernetes API] api --> store[(etcd stores state)] api --> controllers[Controllers watch objects] controllers --> nodes[Nodes run Pods] nodes --> status[Status, events, logs, metrics] status --> api api --> user
2. What Kubernetes Is
Kubernetes is a portable, extensible platform for containerized workloads and services. The key difference from a simple container launcher is that Kubernetes manages not only processes, but API objects, relationships, policies, and control loops.
Kubernetes mainly gives you the following capabilities:
| Capability | Meaning in the workshop | Typical objects |
|---|---|---|
| Declarative configuration | You describe what should run, not every individual step. | Deployment, Service, ConfigMap |
| Automation | Controllers reconcile drift between desired and actual state. | ReplicaSet, Job, HPA |
| Scheduling | Pods are placed on Nodes based on resources and rules. | Pod, Node, PriorityClass |
| Service discovery | Short-lived Pods receive stable reachability. | Service, EndpointSlice, DNS |
| Rollout and rollback | Changes are rolled out in a controlled way and can be reverted. | Deployment, ReplicaSet |
| Self-healing | Failed Pods are restarted or replaced. | Pod, ReplicaSet, kubelet |
| Configuration | Runtime configuration is separated from the image. | ConfigMap, Secret |
| Persistence | Storage can be managed independently from the Pod lifecycle. | PVC, PV, StorageClass |
| Permissions and policies | Access and runtime behavior are constrained. | RBAC, NetworkPolicy, Pod Security |
| Extensibility | Custom APIs and controllers can be added. | CRD, Webhook, Operator |
Important: Kubernetes manages containers through Pods. In Kubernetes, you usually do not work directly with individual containers. You work with resources that create, observe, and replace Pods.
3. What Kubernetes Is Not
Kubernetes is powerful, but it is not a complete application platform without additional decisions. Many misunderstandings occur because Kubernetes is confused with adjacent tools.
| Kubernetes is not ... | Consequence |
|---|---|
| a build system | Images are built outside the cluster, for example in CI/CD. |
| a container registry | Images live in registries such as Harbor, GHCR, ECR, GAR, ACR, or Docker Hub. |
| a programming language | Kubernetes starts and controls workloads, but it does not fix application logic. |
| a database engine | Stateful workloads still require database, backup, and restore design. |
| a replacement for monitoring | Kubernetes provides status and events, but you still need metrics, logs, traces, and alerts. |
| an automatic security guarantee | Defaults can be too permissive; you need RBAC, policies, and hardening. |
| a complete multi-tenancy product | Namespaces help, but they do not replace a proper tenant architecture. |
| a cloud provider | Load balancers, volumes, DNS, and IAM often depend on the provider or add-ons. |
Rule of thumb: Kubernetes provides the control plane. Production readiness comes from architecture decisions, operational processes, security boundaries, and observability.
4. The Mental Model
To understand Kubernetes, do not start by thinking about YAML. YAML is only one representation. The actual model consists of API objects, controllers, status, and events.
flowchart TD manifest[Manifest in YAML or JSON] --> request[API request] request --> admission[Admission: validate and mutate] admission --> storage[(etcd)] storage --> watch[Watch streams] watch --> controller[Controller] controller --> action[Action: create Pod, update Service, bind Volume] action --> runtime[Runtime on Nodes] runtime --> observed[Observed state] observed --> status[Status in the API object]
The most important layers are:
- Intent: You express the desired state in
spec. - Persistence: The API Server stores the object in etcd.
- Observation: Controllers observe object changes through watches.
- Reconciliation: Controllers compare desired and actual state.
- Action: Controllers create, update, or delete dependent objects.
- Feedback: Components write status, conditions, and events.
This leads to three basic rules:
- You always debug Kubernetes through objects, relationships, and status.
- A successful
applyonly means that the API Server accepted the object; it does not automatically mean the workload is running. - If Kubernetes “does nothing,” there is usually a missing controller, a missing permission, missing infrastructure, or an unsatisfiable desired state.
5. Cluster Architecture
A Kubernetes cluster consists of a Control Plane and one or more Worker Nodes. The Control Plane makes global decisions and manages cluster state. Worker Nodes run workloads.
flowchart TB
subgraph cp[Control Plane]
apiserver[kube-apiserver]
etcd[(etcd)]
scheduler[kube-scheduler]
cm[kube-controller-manager]
ccm[cloud-controller-manager]
end
subgraph node1[Worker Node]
kubelet1[kubelet]
runtime1[Container Runtime]
proxy1[kube-proxy or eBPF]
cni1[CNI]
pods1[Pods]
end
subgraph node2[Worker Node]
kubelet2[kubelet]
runtime2[Container Runtime]
proxy2[kube-proxy or eBPF]
cni2[CNI]
pods2[Pods]
end
apiserver <--> etcd
apiserver --> scheduler
apiserver --> cm
apiserver --> ccm
kubelet1 <--> apiserver
kubelet2 <--> apiserver
kubelet1 --> runtime1 --> pods1
kubelet2 --> runtime2 --> pods2
5.1 Control Plane
The Control Plane is the cluster's management layer. In production environments, it should be highly available, monitored, and upgraded in a controlled way.
| Component | Responsibility | Why you need to understand it |
|---|---|---|
kube-apiserver |
Exposes the Kubernetes API. | Every action goes through this API: kubectl, controllers, scheduler, kubelet, and add-ons. |
etcd |
Stores cluster state consistently. | Without etcd integrity, cluster state is at risk. Backup and restore are central. |
kube-scheduler |
Selects Nodes for Pods that have not yet been assigned. | Pending issues are often related to scheduling rules, resources, or taints. |
kube-controller-manager |
Runs core controllers. | Deployments, ReplicaSets, Nodes, Jobs, and other resources are reconciled here. |
cloud-controller-manager |
Connects Kubernetes to cloud-specific APIs. | Load balancers, Node addresses, and volumes may be provider-dependent. |
The API Server is the front door. It authenticates clients, authorizes actions, runs admission steps, validates objects, and persists accepted changes.
etcd is not just a cache. It is the authoritative data store of the Kubernetes API. An etcd backup without a restore test is only an assumption in operations.
The Scheduler does not start containers. It writes a Node assignment into the Pod object. The kubelet on the selected Node takes over from there.
Controllers perform the actual reconciliation. For example, a Deployment controller ensures that ReplicaSets and Pods match the desired Deployment specification.
5.2 Worker Nodes
Worker Nodes run your workloads. Each Node runs components that start Pods, monitor them, connect them to the network, and report back to the Control Plane.
| Component | Responsibility | Typical failure patterns |
|---|---|---|
kubelet |
Node agent; starts and monitors Pods through the runtime. | Node NotReady, Pods do not start, Volume mounts fail. |
| Container Runtime | Runs containers, usually containerd or CRI-O. | ImagePullBackOff, container does not start, runtime socket broken. |
| CNI plugin | Sets up Pod networking. | Pod remains in ContainerCreating, no Pod IP, DNS or routing broken. |
kube-proxy or eBPF dataplane |
Implements Service traffic. | Service appears reachable, but traffic does not reach backends. |
| Node OS | Kernel, cgroups, networking, memory, filesystem. | NodePressure, OOMKills, DiskPressure, kernel or DNS issues. |
The kubelet is not a general-purpose process manager for the entire server. It only manages Pods assigned to it by the Kubernetes API or locally defined as Static Pods.
5.3 Add-ons and Platform Services
Many capabilities that feel built in during daily work actually come from add-ons:
| Area | Typical add-ons | Purpose |
|---|---|---|
| DNS | CoreDNS | Service discovery through DNS names. |
| Networking | Cilium, Calico, Flannel, Antrea | Pod networking, policies, routing, sometimes eBPF dataplane. |
| Ingress | nginx-ingress, HAProxy, Traefik, Contour | HTTP/HTTPS entry into the cluster. |
| Gateway API | Envoy Gateway, Istio, Kong, Cilium Gateway | Role-oriented routing with GatewayClass, Gateway, and Routes. |
| Storage | CSI drivers | Dynamic volumes and storage integration. |
| Observability | Prometheus, Grafana, Loki, Tempo, OpenTelemetry | Metrics, logs, traces, and dashboards. |
| Certificates | cert-manager | Automated certificate management. |
| GitOps | Argo CD, Flux | Synchronization of Git state into the cluster. |
| Policy | Kyverno, Gatekeeper, ValidatingAdmissionPolicy | Validation and enforcement of standards. |
Kubernetes defines interfaces. Add-ons provide concrete implementations. This is why the same manifest can behave differently on two clusters if their CNI, Ingress controller, CSI driver, or admission policies differ.
5.4 High Availability
In learning environments, a cluster often consists of a single Node or a small local simulation. In production, you need different questions:
- Are there multiple API Server instances?
- Is etcd redundant and backed up regularly?
- Are there multiple Worker Nodes across failure zones?
- Are critical add-ons replicated?
- Are PodDisruptionBudgets defined so maintenance does not remove too many Pods at once?
- Are Ingress, DNS, certificate, and storage components themselves highly available?
- Is it clear which component is responsible during a zone, Node, or network failure?
High availability is not just a setting. Reliable availability comes from redundancy, scheduling rules, data management, network design, upgrades, testing, and operational discipline.
6. API and Object Model
Kubernetes is API-centered. Everything you manage in the cluster is either an API object or directly connected to API objects.
6.1 Resources, Kinds, and API Groups
The Kubernetes API is resource-based. You work with resource types such as pods, deployments, services, or secrets. Each object has a kind, for example Pod, Deployment, Service, or Secret.
| Term | Example | Meaning |
|---|---|---|
| API group | apps, batch, rbac.authorization.k8s.io |
Domain namespace for API types. |
| Version | v1, v1beta1, v1alpha1 |
Stability and schema information for the API. |
| Resource | deployments |
URL-oriented plural form in the API. |
| Kind | Deployment |
Type of an individual object. |
| Object | Deployment/shop-api |
Concrete instance of a resource. |
| Subresource | status, scale, logs, exec |
Specialized API path with separate permissions. |
Example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api
namespace: workshop
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: shop-api
template:
metadata:
labels:
app.kubernetes.io/name: shop-api
spec:
containers:
- name: app
image: registry.example.com/platform/shop-api:1.4.2
Here, apps/v1 means that the object belongs to the API group apps in version v1. The kind is Deployment. In the API, you would find this object as the resource deployments in the namespace workshop.
6.2 apiVersion, kind, metadata, spec, status
Almost every Kubernetes object follows a similar base structure.
| Field | Who sets it? | Meaning |
|---|---|---|
apiVersion |
you | API group and version of the object. |
kind |
you | Object type. |
metadata |
you and the system | Name, namespace, labels, annotations, UID, owner references, generation. |
spec |
you | Desired state. |
status |
system | Observed state. |
The most important distinction is spec versus status:
specdescribes what you want.statusdescribes what Kubernetes observes.
Example:
spec:
replicas: 3
status:
replicas: 3
availableReplicas: 2
unavailableReplicas: 1
This object says: you want three replicas. Kubernetes currently observes only two available replicas. The difference is not a YAML error; it is an operational state that needs investigation.
6.3 Namespaces and Cluster-scoped Resources
Namespaces group namespaced resources. A Namespace is an organizational and isolation mechanism, but not a complete security boundary.
Typical namespaced resources:
- Pod
- Deployment
- StatefulSet
- DaemonSet
- Job
- CronJob
- Service
- ConfigMap
- Secret
- Role
- RoleBinding
- ResourceQuota
- LimitRange
- NetworkPolicy
Typical cluster-scoped resources:
- Node
- Namespace
- PersistentVolume
- StorageClass
- ClusterRole
- ClusterRoleBinding
- CustomResourceDefinition
- PriorityClass
- MutatingWebhookConfiguration
- ValidatingWebhookConfiguration
Rule: a resource name plus kind must be unique within its namespace. Cluster-scoped resources do not have a namespace.
Namespaces do not automatically solve the following topics:
- Network isolation: you need NetworkPolicies and a CNI with policy support.
- Resource limits: you need ResourceQuotas and LimitRanges.
- Permission boundaries: you need RBAC.
- Security profiles: you need Pod Security Admission or other admission policies.
- Cost allocation: you need labels, quotas, reporting, and governance.
6.4 Names, UIDs, and Name Generation
Every object has a name and a UID.
| Field | Property |
|---|---|
metadata.name |
set by you or generated; unique within the relevant scope. |
metadata.generateName |
prefix from which Kubernetes generates a name. |
metadata.uid |
system-generated, cluster-wide unique identity of the object. |
metadata.resourceVersion |
stored-object version for watches and concurrent changes. |
metadata.generation |
increases when desired state changes. |
Important: a new object with the same name after deletion has a new UID. For controllers and garbage collection, the identity of the object matters, not just its name.
6.5 Labels, Selectors, and Recommended Labels
Labels are key-value pairs used to organize and select objects. Labels are machine-readable identity and grouping attributes.
Example:
metadata:
labels:
app.kubernetes.io/name: shop-api
app.kubernetes.io/instance: shop-api-prod
app.kubernetes.io/version: "1.4.2"
app.kubernetes.io/component: backend
app.kubernetes.io/part-of: shop
environment: prod
team: platform
Selectors choose objects based on labels:
selector:
matchLabels:
app.kubernetes.io/name: shop-api
Labels are critical for:
- Services that find matching Pods;
- Deployments and ReplicaSets that manage their Pods;
- NetworkPolicies that apply traffic rules to Pod groups;
- observability, cost allocation, and inventory;
- maintenance and platform processes.
A good label strategy:
| Label | Purpose |
|---|---|
app.kubernetes.io/name |
Name of the application or component. |
app.kubernetes.io/instance |
Concrete installation or release instance. |
app.kubernetes.io/version |
Application version. |
app.kubernetes.io/component |
Role within the application, for example frontend, backend, worker. |
app.kubernetes.io/part-of |
Parent system. |
app.kubernetes.io/managed-by |
Tool managing the resource, for example Helm, Kustomize, ArgoCD. |
team |
Responsible team. |
environment |
Environment, for example dev, stage, prod. |
Common misconception: labels are not decoration. Incorrect labels can cause Services to have no Endpoints, ReplicaSets to adopt the wrong Pods, or NetworkPolicies to fail to apply.
6.6 Annotations
Annotations are also key-value pairs, but they are not intended for object selection. Use them for additional metadata, tool configuration, or integration hints.
Examples:
metadata:
annotations:
description: "Backend API for the shop"
owner: "team-platform"
checksum/config: "sha256:..."
Typical uses:
- checksums of ConfigMaps or Secrets to trigger rollouts;
- tool-specific hints for Ingress controllers, service meshes, or backup systems;
- documentation and ownership information;
- policy exceptions, if your governance model allows them.
Rule: if it selects, filters, or groups, it belongs in labels. If it is only additional information, it belongs in annotations.
6.7 Owner References, Garbage Collection, and Finalizers
Owner references describe ownership relationships between objects. A Deployment owns ReplicaSets; a ReplicaSet owns Pods. These relationships enable garbage collection.
flowchart TD deployment[Deployment] --> rs1[ReplicaSet version A] deployment --> rs2[ReplicaSet version B] rs1 --> pod1[Pod] rs1 --> pod2[Pod] rs2 --> pod3[Pod]
When an owner is deleted, Kubernetes may delete dependent objects as well. The exact behavior depends on the deletion strategy:
| Deletion behavior | Meaning |
|---|---|
| Background | The owner is removed; dependent objects are cleaned up afterward. |
| Foreground | The owner remains visible until dependent objects are deleted. |
| Orphan | Dependent objects remain without an owner. |
Finalizers prevent an object from being fully deleted immediately. They give controllers time to clean up external resources.
Typical examples:
- a storage controller removes an external volume;
- a cloud controller removes an external load balancer;
- an operator deletes dependent database resources;
- a backup or policy controller enforces cleanup before deletion.
If an object remains in Terminating for a long time, a finalizer is often involved. Do not remove finalizers blindly. First determine which controller has not finished and which external resource is affected.
6.8 Conditions, Events, and Status
Status is not just a free-text field. Many resources use conditions to express state in a structured way.
Example condition:
status:
conditions:
- type: Available
status: "False"
reason: MinimumReplicasUnavailable
message: Deployment does not have minimum availability.
| Element | Meaning |
|---|---|
type |
Type of condition, for example Available, Progressing, Ready. |
status |
True, False, or Unknown. |
reason |
Machine-readable reason. |
message |
Human-readable description. |
lastTransitionTime |
Time of the most recent status transition. |
Events are time-based hints about decisions and failures. They are especially important for scheduling, image pulls, volume mounts, and probe failures.
Troubleshooting rule: do not read only kubectl get. Check describe, conditions, events, and logs.
7. Declarative Model and Reconciliation
7.1 Desired State and Actual State
Kubernetes works with desired state and observed state.
| Term | Meaning |
|---|---|
| Desired State | State you describe in spec. |
| Actual State | State of the real world: Nodes, processes, network, volumes. |
| Observed State | State Kubernetes currently observes and writes into status. |
| Drift | Difference between desired and actual state. |
| Reconciliation | Continuous alignment performed by controllers. |
Example:
spec:
replicas: 3
If a Node fails and a Pod disappears, a controller detects the drift and creates a replacement. Kubernetes does not, however, “fix” the application itself. If your process crashes because of an invalid configuration, Kubernetes can restart it, but the root cause remains.
7.2 Controller Loop
A controller is a control loop:
- Watch API objects.
- Determine desired state.
- Determine current state.
- Decide what change is required.
- Execute the change.
- Write status back.
- Repeat.
flowchart LR watch[Watch objects] --> compare[Compare desired and actual] compare --> decide[Decide action] decide --> act[Change dependent objects or infrastructure] act --> status[Update status] status --> watch
Controllers should be idempotent. This means: if they see the same situation multiple times, they do not create the same effect repeatedly; instead, they converge the state in a stable way.
7.3 Imperative, Declarative, and GitOps
You can operate Kubernetes imperatively or declaratively.
| Operating style | Example | Suitable for |
|---|---|---|
| Imperative | kubectl create deployment ... |
Learning, quick tests, debugging. |
| Declarative | kubectl apply -f deployment.yaml |
Reproducible configuration. |
| GitOps | Git repository is the authoritative source. | Team and production operations. |
In the workshop, imperative work is useful because it gives you fast feedback loops. For reproducible environments, however, you should work declaratively.
GitOps does not only mean “YAML in Git.” GitOps means:
- Git contains the desired state;
- a controller synchronizes that state into the cluster;
- changes go through review, commit, and an audit trail;
- drift is detected or corrected;
- rollbacks happen through Git history and controlled synchronization.
7.4 Server-side Apply and Field Ownership
When working declaratively, it matters who manages which field. Server-side Apply stores so-called field ownership. This allows Kubernetes to determine which tool or manager set a field.
This becomes relevant when multiple actors modify the same object:
- a GitOps controller manages the Deployment;
- an HPA manages the replica count through the
scalesubresource; - an admission webhook adds defaults;
- an operator sets status or dependent fields.
Conflicts occur when two managers both want to manage the same field authoritatively. The concept protects against unnoticed overwrites.
8. Pods
8.1 A Pod as a Logical Host
A Pod is the smallest deployable unit in Kubernetes. It is not a single container, but a runtime environment for one or more containers.
Containers in a Pod share:
- network namespace;
- IP address;
- port space;
- optional shared volumes;
- lifecycle placement on the same Node.
A Pod models a logical host. Inside a Pod, containers communicate via localhost. From the outside, the Pod is reached through its Pod IP while it exists.
Important: Pods are fundamentally short-lived. You should not expect a specific Pod to exist permanently. Workload controllers create replacements when Pods disappear.
8.2 Containers in a Pod
The most common case is a Pod with exactly one application container:
apiVersion: v1
kind: Pod
metadata:
name: web
namespace: workshop
spec:
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 80
Multiple containers in one Pod are useful when they are tightly coupled and must be scheduled together. Examples:
- a sidecar collects logs or exports metrics;
- a proxy mediates traffic for the main application;
- a helper container updates files in a shared volume;
- an adapter translates protocols.
Common misconception: multiple containers in a Pod are not a substitute for scaling. If you need more capacity, you usually increase the number of Pods, not the number of identical containers inside one Pod.
8.3 Init Containers, Sidecars, and Ephemeral Containers
| Type | Purpose | Lifecycle |
|---|---|---|
| Init Container | Preparation before the app starts. | Runs to completion before app containers start. |
| Sidecar Container | Supporting long-running function next to the app. | Runs in parallel with the app. |
| Ephemeral Container | Debugging in running Pods. | Injected after creation, not intended for regular operation. |
Init Containers are suitable for tasks such as:
- waiting for dependent services;
- one-time file initialization;
- migration or setup checks;
- separating tooling that should not be included in the app image.
Ephemeral Containers are valuable for debugging when your app image is intentionally minimal and does not contain a shell or diagnostic tools.
8.4 Pod Lifecycle
Pods move through phases. The phase is a broad summary, not a complete error report.
| Phase | Meaning |
|---|---|
Pending |
Pod was accepted but has not fully started. Scheduling, image pull, or volume mount may still be pending. |
Running |
Pod is bound to a Node, and at least one container is running or starting. |
Succeeded |
All containers terminated successfully and will not be restarted. |
Failed |
At least one container terminated with an error, and the Pod will not continue running. |
Unknown |
Node state cannot be determined reliably. |
There are also container states:
| Container state | Meaning |
|---|---|
Waiting |
Container is not running yet, for example because of image pull or backoff. |
Running |
Container is running. |
Terminated |
Container has exited. |
Typical waiting reasons:
ContainerCreatingErrImagePullImagePullBackOffCrashLoopBackOffCreateContainerConfigErrorCreateContainerError
8.5 Restart Policies
The restartPolicy describes when containers in a Pod are restarted.
| Policy | Meaning | Typical use |
|---|---|---|
Always |
Container is restarted when it exits. | Deployments, DaemonSets, StatefulSets. |
OnFailure |
Restart only after failure exit status. | Jobs. |
Never |
No restart. | One-off test or debug Pods. |
For Deployments, Always is standard. Jobs usually use OnFailure or Never, because they represent tasks that terminate.
8.6 Probes
Probes are diagnostics executed periodically by the kubelet.
| Probe | Question | Effect on failure |
|---|---|---|
| Startup Probe | Has the application finished initial startup? | While it fails, liveness and readiness evaluation is deferred. |
| Readiness Probe | May this Pod receive traffic? | Pod is removed from Service endpoints. |
| Liveness Probe | Does the container need to be restarted? | kubelet restarts the container. |
Example:
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
Good probe rules:
- Readiness checks dependencies that are relevant for serving traffic.
- Liveness checks only whether a restart is likely to improve the situation.
- Startup probes protect slow-starting applications from premature liveness restarts.
- Probes must not be expensive.
- A broken database connection is not automatically a liveness failure; otherwise you restart the application repeatedly during an external outage.
8.7 Graceful Shutdown
When a Pod is terminated, Kubernetes applies a termination grace period. The kubelet sends a termination signal to the container process. The application should handle this signal and shut down cleanly.
Important points:
- The application should stop accepting new requests.
- In-flight requests should complete or be aborted in a controlled way.
- Readiness should switch to
Falseearly. - Background jobs should have checkpoints or safe stop points.
- The grace period must match the application.
preStophooks can help, but they do not replace proper signal handling.
Common misconception: Kubernetes can terminate a process, but it cannot automatically establish transactional application consistency.
8.8 Common Pod Failure Patterns
| Failure pattern | Conceptual cause | What to check |
|---|---|---|
Pending |
No suitable Node, missing resources, taints, PVC not bound. | Events, scheduler messages, requests, Node capacity, tolerations. |
ImagePullBackOff |
Image unavailable or missing registry permission. | Image name, tag, registry, ImagePullSecrets, network. |
CrashLoopBackOff |
Process exits repeatedly. | Logs, exit code, config, Secrets, liveness probe. |
CreateContainerConfigError |
Pod specification references a missing ConfigMap or Secret. | Env refs, volume refs, namespaces. |
OOMKilled |
Memory limit exceeded. | Limits, memory profile, leaks, in-memory emptyDir. |
ContainerCreating stays active |
Runtime, CNI, or volume mount is blocked. | Events, kubelet, CNI, CSI, Node state. |
9. Workloads and Controllers
Pods are rarely created directly. You use workload resources so Kubernetes can replicate, replace, update, and monitor Pods.
flowchart TD deployment[Deployment] --> replicaset[ReplicaSet] replicaset --> pods[Pods] statefulset[StatefulSet] --> spods[Pods with stable identity] daemonset[DaemonSet] --> dpods[Pod per eligible Node] cronjob[CronJob] --> job[Job] job --> jpods[Pods until completion]
9.1 ReplicaSet
A ReplicaSet keeps a stable number of identical Pods available. It uses a selector and a Pod template.
In practice, you rarely create ReplicaSets directly. A Deployment usually creates ReplicaSets for you.
Important:
- A ReplicaSet controls replication.
- It does not provide a rich rollout history like a Deployment.
- It can select Pods based on labels.
- Poorly chosen selectors can capture unrelated Pods.
9.2 Deployment
A Deployment is the standard resource for stateless applications. It manages ReplicaSets and enables declarative rollouts.
A Deployment is suitable when:
- Pods are interchangeable;
- state lives outside the Pod;
- horizontal scaling is possible;
- rolling updates and rollbacks are required;
- the Service can distribute traffic to any healthy replica.
Example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api
namespace: workshop
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
app.kubernetes.io/name: shop-api
template:
metadata:
labels:
app.kubernetes.io/name: shop-api
spec:
containers:
- name: app
image: registry.example.com/platform/shop-api:1.4.2
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /ready
port: http
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 256Mi
Important fields:
| Field | Meaning |
|---|---|
replicas |
Desired number of Pods. |
selector |
Which Pods belong to the Deployment. |
template |
Pod template for new Pods. |
strategy |
Rollout behavior. |
revisionHistoryLimit |
Number of old ReplicaSets kept for rollback history. |
progressDeadlineSeconds |
Time limit for rollout progress. |
Common misconception: a Deployment is not suitable for every stateful workload. If Pod identity or volume assignment must be stable, evaluate a StatefulSet.
9.3 StatefulSet
A StatefulSet manages Pods with stable identity. The Pods are not fully interchangeable.
StatefulSets are suitable for applications with:
- stable names;
- stable network identity;
- stable volume assignment;
- ordered creation and scaling;
- stateful roles.
Example Pod names:
postgres-0
postgres-1
postgres-2
In a StatefulSet, postgres-0 keeps its identity even if the Pod is rescheduled. A Deployment would instead create interchangeable Pod names.
Important concepts:
| Concept | Meaning |
|---|---|
| Headless Service | Stable DNS names for individual Pods. |
volumeClaimTemplates |
A dedicated PVC is created for each Pod. |
| Ordinal | Number in the Pod name, for example 0, 1, 2. |
| OrderedReady | Default behavior for ordered creation and scaling. |
Common misconception: StatefulSet does not automatically mean database safety. Backup, restore, replication, quorum, data migration, and storage classes remain application and platform concerns.
9.4 DaemonSet
A DaemonSet ensures that a Pod runs on all or selected Nodes.
Typical uses:
- log agent on every Node;
- monitoring agent;
- CNI component;
- storage node plugin;
- security agent;
- node-local proxy.
DaemonSets react to Node additions and removals. When a new eligible Node joins, the DaemonSet controller creates a Pod there.
Important:
- DaemonSet Pods are often infrastructure-critical.
- DaemonSet Pods often require special permissions or host access.
- DaemonSet Pods should be reviewed and hardened separately.
- During Node drain, remember that DaemonSet Pods are usually not evicted like ordinary workload Pods.
9.5 Job
A Job runs a task to completion. A Job creates Pods and tracks successful completions.
Typical uses:
- data migration;
- batch processing;
- one-time import;
- report generation;
- cleanup task;
- test run inside the cluster.
Important fields:
| Field | Meaning |
|---|---|
completions |
Number of successful completions. |
parallelism |
Number of Pods running in parallel. |
backoffLimit |
Number of failed attempts before the Job is marked failed. |
activeDeadlineSeconds |
Maximum runtime. |
ttlSecondsAfterFinished |
Automatic cleanup after completion. |
Common misconception: a Job is not automatically idempotent. Your task must handle Pod restarts or repeated Job execution itself.
9.6 CronJob
A CronJob creates Jobs on a schedule. The syntax follows cron.
Typical uses:
- regular backups;
- periodic reports;
- scheduled synchronization;
- cleanup of old data;
- health or consistency checks.
Important fields:
| Field | Meaning |
|---|---|
schedule |
Cron expression. |
timeZone |
Time zone, if used and supported. |
concurrencyPolicy |
Handling of overlapping runs: Allow, Forbid, Replace. |
startingDeadlineSeconds |
Catch-up window for missed starts. |
successfulJobsHistoryLimit |
Number of successful Jobs retained. |
failedJobsHistoryLimit |
Number of failed Jobs retained. |
Common misconception: CronJob scheduling does not replace business logic. Tasks must handle duplicate starts, partially failed steps, and external dependencies.
9.7 HorizontalPodAutoscaler
The HorizontalPodAutoscaler scales the number of Pods for a workload based on metrics.
Typical sources:
- CPU utilization;
- memory utilization;
- custom metrics;
- external metrics.
Prerequisites:
- requests must be set meaningfully;
- Metrics API or a suitable metric source must be available;
- the application must be horizontally scalable;
- startup time, warm-up, and load distribution must be considered.
Common misconception: HPA does not solve a capacity problem if the cluster has no free resources. In that case, you need cluster autoscaling, larger Nodes, or less load.
9.8 PodDisruptionBudget
A PodDisruptionBudget limits voluntary disruptions. It helps during Node drain, upgrades, or planned maintenance.
Example:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: shop-api
namespace: workshop
spec:
minAvailable: 2
selector:
matchLabels:
app.kubernetes.io/name: shop-api
Important:
- A PDB does not protect against Node crashes or hardware failures.
- Overly strict PDBs can block maintenance.
- PDBs require enough replicas to be meaningful.
- PDBs must align with rollout strategy and availability targets.
10. Networking, Services, and Access
Kubernetes networking consists of several layers. You need to distinguish whether you are talking about Pod-to-Pod communication, service discovery, external access, or policy isolation.
flowchart LR client[External client] --> lb[Load balancer] lb --> ingress[Ingress Controller or Gateway] ingress --> service[Service] service --> eps[EndpointSlices] eps --> pod1[Pod A] eps --> pod2[Pod B] pod1 <--> pod2
10.1 Pod Network
Basic assumptions of Kubernetes networking:
- Every Pod receives its own IP address.
- Pods can generally communicate Pod-to-Pod if the network and policies allow it.
- Containers in the same Pod share an IP and reach each other via
localhost. - Services provide stable virtual reachability.
The Pod IP is not stable. When a Pod is replaced, the new Pod usually receives a different IP. Applications should not rely on Pod IPs permanently.
10.2 CNI
CNI stands for Container Network Interface. Kubernetes uses CNI plugins to set up Pod networking.
Typical responsibilities of a CNI plugin:
- assign Pod IPs;
- create network interfaces in the Pod namespace;
- provide routing or overlay networking;
- enforce NetworkPolicies;
- optionally provide an eBPF dataplane;
- optionally support encryption, observability, or service mesh integration.
Important: NetworkPolicy objects only have an effect if your network plugin enforces them. A cluster can accept NetworkPolicies without actually enforcing them if the CNI does not support policy enforcement.
10.3 Service
A Service decouples clients from short-lived Pods. It typically selects matching Pods through labels and provides a stable address.
Example:
apiVersion: v1
kind: Service
metadata:
name: shop-api
namespace: workshop
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: shop-api
ports:
- name: http
port: 80
targetPort: http
Terms:
| Field | Meaning |
|---|---|
selector |
Selects Pods based on labels. |
port |
Service port. |
targetPort |
Port in the target Pod; can be a name or number. |
protocol |
TCP, UDP, or SCTP. |
clusterIP |
Virtual internal Service IP. |
Common misconception: a Service does not start Pods. If a Service has no Endpoints, the cause is usually labels, selectors, readiness, or missing Pods.
10.4 EndpointSlices
EndpointSlices describe the actual backends of a Service. EndpointSlices replace or complement classic Endpoints for scalability and modern functionality.
EndpointSlices contain information such as:
- target addresses;
- ports;
- protocols;
- ready state;
- Node and zone hints.
When debugging Service problems, the chain matters:
Service selector -> Pod labels -> Pod readiness -> EndpointSlice -> Traffic
10.5 Service Types
| Type | Reachability | Use |
|---|---|---|
ClusterIP |
internal to the cluster | Default for internal communication. |
NodePort |
port on every Node | Simple exposure, often a building block for LoadBalancer. |
LoadBalancer |
external load balancer | Cloud or platform integration. |
ExternalName |
DNS CNAME to an external name | Abstraction of external services. |
| Headless Service | no ClusterIP | Direct DNS resolution to Pod endpoints, often for StatefulSets. |
LoadBalancer does not work magically. It needs an implementation: cloud controller, MetalLB, Cilium, kube-vip, or another platform component.
10.6 DNS
CoreDNS provides DNS-based service discovery in many clusters. Typical Service names:
shop-api
shop-api.workshop
shop-api.workshop.svc
shop-api.workshop.svc.cluster.local
Pods normally receive DNS configuration that lets them resolve Services inside the cluster.
Typical DNS problems:
- CoreDNS Pods are not running;
- network between Pods and CoreDNS is blocked;
- NetworkPolicy blocks DNS traffic;
- Node DNS or upstream DNS is broken;
- wrong namespace is assumed;
- Service has no Endpoints;
- application caches DNS too aggressively.
10.7 Ingress
Ingress describes HTTP/HTTPS routing into the cluster. An Ingress object alone is not enough; you need an Ingress controller.
Ingress is suitable for:
- host-based routing;
- path-based routing;
- TLS termination;
- centralized HTTP entry points;
- simple web workloads.
Conceptual flow:
Client -> Load Balancer -> Ingress Controller -> Ingress rule -> Service -> Pods
Common misconception: Ingress is not a Service type. Ingress is its own resource interpreted by a controller.
10.8 Gateway API
Gateway API is a more modern, role-oriented API family for network access and routing. The API family is implemented through CRDs and controllers.
Important resources:
| Resource | Role |
|---|---|
GatewayClass |
Describes a Gateway implementation or class. |
Gateway |
Describes a concrete infrastructure instance or listeners. |
HTTPRoute |
Describes HTTP routing rules. |
GRPCRoute |
Describes gRPC routing rules. |
TCPRoute / TLSRoute |
Additional protocols depending on implementation and support. |
Gateway API separates responsibilities more clearly:
- infrastructure team manages GatewayClasses and Gateways;
- platform team sets global policies;
- application team manages Routes for its own Services.
This is especially useful in larger organizations, because not every application should directly control infrastructure details.
10.9 NetworkPolicies
NetworkPolicies describe allowed traffic at Layer 3 and Layer 4. These policies are application-centric and select Pods through labels.
The default behavior without NetworkPolicy is typically open: Pods are not isolated. Once a NetworkPolicy selects a Pod for ingress or egress, only allowed connections apply for that direction.
Example: default deny for ingress in a Namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: workshop
spec:
podSelector: {}
policyTypes:
- Ingress
Example: allow frontend to backend:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-api
namespace: workshop
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: shop-api
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: shop-frontend
ports:
- protocol: TCP
port: 8080
Important rules:
- NetworkPolicies work only with a supporting CNI.
- Policies are additive: multiple policies can allow traffic together.
- There is no explicit deny rule in the standard model; denial is modeled by the absence of allow rules after isolation.
- DNS must be allowed intentionally when egress is restricted.
- Node traffic and some infrastructure paths may require special exceptions.
10.10 Common Network Failure Patterns
| Failure pattern | Concept area | Diagnostic questions |
|---|---|---|
| Service has no Endpoints | Labels, selector, readiness | Do the Service selector and Pod labels match? Are Pods ready? |
| DNS resolution fails | CoreDNS, network, policy | Is CoreDNS running? Is UDP/TCP 53 allowed? Is the name fully qualified? |
| Ingress returns 404 | Ingress rules, host, path, controller | Does the Host header match? Is the IngressClass handled? |
| LoadBalancer remains pending | Cloud or bare-metal integration | Is there a controller or MetalLB? Are IP pools available? |
| Pod cannot reach external destinations | Egress policy, NAT, Node network | Is a NetworkPolicy blocking traffic? Does Node egress work? |
| Service is reachable but app responds incorrectly | Port mapping, protocol, app | Does targetPort point to the correct container port? |
11. Configuration and Secrets
Configuration should be separated from the image. A container image describes the artifact and default behavior. Environment-specific values belong in Kubernetes objects or external configuration systems.
11.1 ConfigMap
A ConfigMap stores non-secret configuration.
Example:
apiVersion: v1
kind: ConfigMap
metadata:
name: shop-api-config
namespace: workshop
data:
LOG_LEVEL: "info"
FEATURE_CHECKOUT_V2: "true"
application.yaml: |
server:
port: 8080
cache:
ttlSeconds: 300
ConfigMaps are suitable for:
- feature flags without secrets;
- log levels;
- URLs of internal services;
- configuration files;
- non-sensitive parameters.
ConfigMaps are not suitable for:
- passwords;
- tokens;
- private keys;
- customer data;
- confidential configuration.
11.2 Secret
A Secret stores sensitive data as a Kubernetes object. By default, values are base64-encoded, not automatically strongly protected. Base64 is not encryption.
Example:
apiVersion: v1
kind: Secret
metadata:
name: shop-api-secret
namespace: workshop
type: Opaque
stringData:
DB_USER: shop
DB_PASSWORD: replace-me
Good Secret rules:
- do not commit real secrets to Git;
- strictly limit RBAC access to Secrets;
- enable encryption at rest for Secret data;
- evaluate external secret systems such as Vault, cloud-native secret managers, or External Secrets Operator;
- plan secret rotation;
- do not print Secrets in logs;
- avoid exposing Secrets as environment variables if process listings, dumps, or debugging increase the risk.
11.3 Environment Variables and Volumes
ConfigMaps and Secrets can be used as environment variables or mounted as files.
Environment variables:
envFrom:
- configMapRef:
name: shop-api-config
- secretRef:
name: shop-api-secret
Volume mount:
volumes:
- name: config
configMap:
name: shop-api-config
containers:
- name: app
volumeMounts:
- name: config
mountPath: /etc/shop
readOnly: true
Difference:
| Mounting style | Advantage | Limitation |
|---|---|---|
| Environment variable | Simple for classic app configuration. | Changes do not automatically appear in the running process. |
| Volume | Files can be updated. | Application must detect file changes or reload. |
| External secret system | better rotation and central governance. | additional components and permissions. |
11.4 Configuration Changes and Rollouts
A change to a ConfigMap or Secret does not automatically restart every Deployment Pod. Many teams use checksum annotations in the Pod template so that a change to the config object triggers a new ReplicaSet rollout.
Concept:
metadata:
annotations:
checksum/config: "sha256-of-rendered-config"
When the annotation in the Pod template changes, the template changes. The Deployment creates a new ReplicaSet.
Common misconception: a configuration change is a release event. Every configuration change needs validation, rollout strategy, and rollback path just like code.
12. Storage
Container filesystems are short-lived. When a container or Pod is replaced, local state is not reliably durable. Kubernetes therefore separates workload lifecycle and storage lifecycle.
12.1 Ephemeral Storage
Ephemeral storage lives with the Pod or Node.
Examples:
| Type | Lifetime | Use |
|---|---|---|
| Container filesystem | Container lifecycle | temporary process data, not persistent. |
emptyDir |
Pod lifecycle | scratch space, shared files between containers. |
memory-backed emptyDir |
Pod lifecycle, RAM-backed | fast temporary data, but memory limits matter. |
| Logs on Node | depends on runtime and rotation | short-term diagnostics, not long-term archive. |
Important: memory-backed emptyDir volumes can contribute to the Pod's memory usage. Without sensible limits, this can cause Node or Pod problems.
12.2 Volumes
Volumes provide filesystem areas to a Pod. There are many volume types, for example:
emptyDir;configMap;secret;projected;persistentVolumeClaim;- CSI-based volumes.
Volume mounts happen inside containers:
volumes:
- name: data
persistentVolumeClaim:
claimName: shop-data
containers:
- name: app
volumeMounts:
- name: data
mountPath: /var/lib/shop
Separate the following in your design:
- configuration data;
- secret data;
- temporary working data;
- persistent application data;
- logs and telemetry;
- backups.
12.3 PersistentVolume, PersistentVolumeClaim, and StorageClass
The PersistentVolume subsystem abstracts storage provisioning from storage consumption.
| Object | Perspective | Meaning |
|---|---|---|
| PersistentVolume | Cluster/storage perspective | A provisioned storage volume in the cluster. |
| PersistentVolumeClaim | Workload perspective | A storage request by an application. |
| StorageClass | Provisioning profile | Defines dynamic provisioning and parameters. |
Flow:
flowchart LR pvc[PVC requests storage] --> sc[StorageClass] sc --> csi[CSI provisioner] csi --> pv[PV is created] pv --> pod[Pod mounts Volume via PVC]
Example PVC:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shop-data
namespace: workshop
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
Important storage questions:
- Which
accessModeis required? - Is the volume bound to a Node or zone?
- What performance does the StorageClass guarantee?
- What happens when the PVC is deleted?
- Are snapshots available?
- Has restore been tested?
- Does the driver support expansion?
- How is data encrypted?
12.4 CSI
CSI stands for Container Storage Interface. It is the standardized interface through which storage systems integrate with Kubernetes.
Typical CSI components:
- controller component for provisioning, attach, detach, snapshot, or expansion;
- node component as a DaemonSet for mounts on Nodes;
- StorageClass parameters for driver behavior;
- VolumeSnapshotClass for snapshots, if supported.
CSI separates Kubernetes core logic from storage-specific implementation. That is powerful, but it also means details depend heavily on the specific driver.
12.5 Stateful Storage Design
For stateful workloads, PVCs alone are not enough. You need a complete data concept.
Review questions:
- Is the application suitable for Kubernetes operations?
- Can the application tolerate Pod restarts?
- Is storage zonal or regional?
- Must the Pod return to the same Node or zone?
- How does replication work?
- How does quorum work?
- How does backup work?
- How does restore into a new environment work?
- Are migration steps idempotent?
- What happens during split brain or network partitions?
Common misconception: “StatefulSet plus PVC” is not a complete database architecture.
12.6 Backup, Restore, and Data Consistency
Backup concepts must account for data consistency.
| Layer | Example | Risk |
|---|---|---|
| Volume snapshot | CSI snapshot | May be inconsistent without application coordination. |
| Database dump | pg_dump, mysqldump |
Can be large or slow, but logically consistent. |
| Replication-based | Database replication | does not automatically protect against logical corruption. |
| Cluster backup | etcd backup, resource export | does not automatically back up external volumes. |
Restore is more important than backup. A backup is reliable only after you regularly test the restore process.
13. Scheduling and Resources
13.1 Scheduling Fundamentals
The Scheduler assigns Pods to Nodes. It considers, among other things:
- resource requests;
- Node capacity and allocatable resources;
- NodeSelector;
- Node affinity;
- Pod affinity and anti-affinity;
- taints and tolerations;
- topology spread constraints;
- volume bindings;
- priority and preemption;
- policies and scheduler profiles.
Scheduling is a placement decision, not proof that the application can start. A Pod can be successfully scheduled and still fail later because of image pull, config, volume mount, or runtime issues.
13.2 Requests and Limits
Requests and limits are core concepts for capacity planning.
| Field | Effect |
|---|---|
| CPU request | Scheduler reserves CPU capacity; CPU weighting under contention. |
| Memory request | Scheduler reserves memory capacity; relevant to eviction priority. |
| CPU limit | limits CPU time; can cause throttling. |
| Memory limit | hard memory boundary; exceeding it can lead to OOMKill. |
| Ephemeral storage request/limit | capacity planning and constraint for local temporary storage. |
Example:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 256Mi
Notes:
- CPU is specified in cores or millicores;
100mequals 0.1 CPU. - Memory is specified in bytes or units such as
MiorGi. - For many applications, a memory limit is more useful than a CPU limit, because CPU throttling can cause latency problems.
- Without requests, the Scheduler cannot plan capacity reliably.
- Overly high requests cause poor utilization or
PendingPods. - Overly low memory limits cause OOMKills.
13.3 QoS Classes
Kubernetes assigns Pods to QoS classes:
| QoS class | Condition | Meaning |
|---|---|---|
Guaranteed |
all containers have equal requests and limits for CPU and memory. | highest protection class under resource pressure. |
Burstable |
at least one request or limit is set, but not fully Guaranteed. | standard for many workloads. |
BestEffort |
no requests or limits set. | lowest priority under resource pressure. |
QoS influences which Pods are more likely to be evicted under Node resource pressure.
13.4 NodeSelector, Affinity, and Anti-affinity
NodeSelector is the simple form of Node selection:
nodeSelector:
kubernetes.io/os: linux
Node affinity is more expressive:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- eu-central-1a
- eu-central-1b
Pod anti-affinity can prevent replicas of the same application from landing on the same Node or in the same zone.
Common misconception: hard rules do not automatically improve availability. Too many hard constraints can make Pods unschedulable.
13.5 Taints and Tolerations
Taints repel Pods. Tolerations allow Pods to be scheduled onto Nodes despite a taint.
Example:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
Typical effects:
| Effect | Meaning |
|---|---|
NoSchedule |
New Pods without matching toleration are not scheduled. |
PreferNoSchedule |
Scheduler tries to avoid placing Pods there. |
NoExecute |
Running Pods without toleration may be removed. |
Taints are useful for dedicated Nodes, GPU Nodes, infrastructure Nodes, or maintenance situations.
13.6 Topology Spread Constraints
Topology spread constraints distribute Pods across topologies such as Nodes, zones, or regions.
Goal: avoid all replicas running on the same Node or in the same zone.
Concept example:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: shop-api
Make sure the topology labels exist and are correct on Nodes.
13.7 PriorityClass and Preemption
PriorityClass assigns priorities to Pods. When resources are scarce, Kubernetes can evict lower-priority Pods so higher-priority Pods can run.
This is powerful and dangerous:
- critical infrastructure can be preferred;
- incorrectly assigned priorities can evict important workloads;
- preemption does not solve persistent undercapacity;
- priorities must be governed organizationally.
13.8 ResourceQuota and LimitRange
ResourceQuotas limit resource usage in a Namespace.
Examples:
- maximum CPU requests;
- maximum memory limits;
- number of Pods;
- number of Services;
- number of PVCs;
- storage consumption through PVCs.
LimitRanges set defaults and boundaries per object or container.
Relationship:
| Object | Responsibility |
|---|---|
| ResourceQuota | limits aggregate usage in the Namespace. |
| LimitRange | sets default requests/limits and min/max values. |
Common misconception: quotas without monitoring create surprises. Teams must be able to see why new workloads are rejected.
13.9 Autoscaling Concepts
Autoscaling exists at several layers:
| Layer | Resource or component | Scales |
|---|---|---|
| HorizontalPodAutoscaler | HPA | number of Pods. |
| VerticalPodAutoscaler | VPA | requests and limits. |
| Cluster Autoscaler | cluster component | number of Nodes. |
| KEDA | add-on | event-driven scaling. |
Autoscaling requires stable signals. CPU-based scaling is simple, but not always aligned with business behavior. Queue length, requests per second, latency, or external metrics may be more meaningful.
14. Security
Kubernetes security is layered work. Enabling one feature is not enough.
flowchart TD authn[Authentication] --> authz[Authorization] authz --> admission[Admission Control] admission --> podsec[Pod Security] podsec --> runtime[Runtime hardening] runtime --> network[Network Policies] network --> secrets[Secret protection] secrets --> audit[Audit and monitoring]
14.1 Authentication, Authorization, and Admission
| Step | Question | Example |
|---|---|---|
| Authentication | Who are you? | Certificate, token, OIDC. |
| Authorization | What may you do? | RBAC. |
| Admission | May this object be stored in this form? | Pod Security, webhooks, ValidatingAdmissionPolicy. |
Admission happens after authentication and authorization, but before the object is persisted. Admission can validate or mutate objects.
Typical admission tasks:
- set default values;
- reject unsafe Pods;
- restrict image registries;
- enforce required labels;
- require resource requests;
- forbid HostPath or privileged Pods;
- inject sidecars.
14.2 ServiceAccounts
A ServiceAccount is an identity for Pods and controllers inside the cluster. Pods can use ServiceAccount tokens to talk to the Kubernetes API.
Rules:
- use dedicated ServiceAccounts per application;
- bind only the permissions that are actually required;
- avoid the
defaultServiceAccount for production workloads; - set
automountServiceAccountToken: falseif the application does not need the Kubernetes API; - check which Pods run with which ServiceAccount.
Example:
apiVersion: v1
kind: ServiceAccount
metadata:
name: shop-api
namespace: workshop
automountServiceAccountToken: false
14.3 RBAC
RBAC controls access to Kubernetes resources. It consists of roles and bindings.
| Object | Scope | Purpose |
|---|---|---|
| Role | Namespace | Permissions within a Namespace. |
| ClusterRole | Cluster-wide or reusable | Permissions for cluster-scoped resources or as a template. |
| RoleBinding | Namespace | Binds a Role or ClusterRole to subjects in the Namespace. |
| ClusterRoleBinding | Cluster-wide | Binds a ClusterRole cluster-wide. |
Subjects can be:
- users;
- groups;
- ServiceAccounts.
Example:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: workshop
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
Good RBAC rules:
- no unnecessary
cluster-admin; - no wildcards without justification;
- protect Secrets separately;
- treat
create pods/execandpods/portforwarddeliberately; - account for subresources;
- audit bindings regularly.
14.4 Pod Security Standards
Pod Security Standards define three profiles:
| Profile | Goal |
|---|---|
| Privileged | No restrictions; for highly trusted infrastructure workloads. |
| Baseline | Prevents known privilege escalations while remaining broadly compatible. |
| Restricted | Strongly hardened profile aligned with current hardening practices. |
Pod Security Admission can evaluate Namespaces based on labels:
metadata:
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Common misconception: Pod Security does not replace RBAC, NetworkPolicy, or image verification. It is a runtime/spec hardening layer.
14.5 SecurityContext
The SecurityContext describes security options at Pod or container level.
Example:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Important options:
runAsNonRoot;runAsUserandrunAsGroup;fsGroup;allowPrivilegeEscalation;privileged;- Linux capabilities;
- seccomp profile;
- AppArmor, depending on environment;
readOnlyRootFilesystem.
Good practice: start with a restrictive profile and add permissions only when justified.
14.6 Images and Supply Chain
Kubernetes starts images, but it does not automatically guarantee their origin, integrity, or security.
Review questions:
- Where does the image come from?
- Is the tag immutable, or do you use digests?
- Is the image scanned?
- Are SBOMs available?
- Can the signature be verified?
- Are base images current?
- Does the image contain unnecessary tools or secrets?
- Does the process run as non-root?
Use immutable references where possible:
image: registry.example.com/platform/shop-api@sha256:0123456789abcdef...
Tags such as latest are unsuitable for reproducible deployments.
14.7 Secrets, Encryption, and External Secret Systems
Secret protection consists of several layers:
- minimize access through RBAC;
- enable encryption at rest;
- monitor Secret access in audits;
- define rotation and expiry;
- integrate external secret systems;
- do not store Secrets in images, logs, or Git;
- restrict debug access.
Important: anyone who can create a Pod that mounts a Secret can often indirectly obtain Secret values. RBAC must therefore consider Pod creation, exec, debug, and Secret access together.
14.8 Tenant Isolation
Namespaces are one building block for tenant isolation, but not a hard boundary like separate clusters.
Tenant isolation typically requires:
- Namespace strategy;
- RBAC per team or application;
- ResourceQuotas;
- LimitRanges;
- NetworkPolicies;
- Pod Security enforcement;
- admission policies;
- separate Ingress/Gateway rules;
- Secret isolation;
- audit and cost reporting;
- potentially separate clusters for strong isolation.
Decision rule: if tenants do not trust each other, a separate cluster is often simpler and safer than very complex isolation inside a shared cluster.
15. Observability and Troubleshooting
Kubernetes provides many signals, but you must know which question you are asking.
15.1 Status, Events, and Logs
| Signal | Question |
|---|---|
| Object status | What does Kubernetes observe about this object? |
| Conditions | Which states are true, false, or unknown? |
| Events | Which decisions or failures occurred? |
| Container logs | What does the application or process report? |
| kubelet logs | What happens on the Node during startup or mounting? |
| Controller logs | What does the responsible controller decide? |
Principle: kubectl get shows symptoms. kubectl describe, events, logs, and status details take you closer to causes.
15.2 Metrics
Metrics answer quantitative questions:
- How much CPU does the Pod use?
- How much memory does the container use?
- How many requests does the application handle?
- How high is latency?
- How many errors occur?
- How full is a volume?
- What is the API Server latency?
- How many etcd leader changes occurred?
Typical metric sources:
- Metrics Server for basic resource metrics;
- Prometheus for time series;
- kube-state-metrics for object states;
- cAdvisor/kubelet metrics for container and Node signals;
- controller and add-on metrics;
- application metrics.
15.3 Traces
Traces help with distributed requests across several services. Kubernetes itself does not generate application traces. You need instrumentation, for example through OpenTelemetry.
Traces are especially useful when:
- several microservices are involved;
- latency paths are unclear;
- errors occur only in specific call chains;
- external dependencies are involved;
- you want to understand SLOs at request level.
15.4 Auditing
Audit logs answer governance questions:
- Who changed which object?
- Who read a Secret?
- Who executed
execinto a Pod? - Which ServiceAccounts talk to the API?
- Which admission decisions were made?
Audit is central for security, compliance, and incident response. Without audit, many cluster changes are difficult to reconstruct after the fact.
15.5 Layered Fault Diagnosis
Diagnose Kubernetes problems layer by layer:
Manifest -> API -> Admission -> Scheduling -> Node -> Runtime -> Network -> Storage -> Application
| Layer | Typical questions |
|---|---|
| Manifest | Is the YAML valid? Are apiVersion, kind, names, and references correct? |
| API | Was the object accepted? Are there admission errors? |
| Scheduling | Can the Scheduler find a suitable Node? |
| Node | Is the Node Ready? Are there pressure conditions or taints? |
| Runtime | Can the image be pulled and the container started? |
| Network | Does the Pod have IP, DNS, and Service reachability? |
| Storage | Are PVCs bound and volumes mounted? |
| Application | Does the process start? Are config, Secrets, and dependencies correct? |
16. Platform Operations
16.1 Versions and Upgrades
Kubernetes evolves continuously. Versions affect API compatibility, feature gates, add-ons, container runtime, CNI, CSI, and operating systems.
Upgrade questions:
- Which Kubernetes version is currently running?
- Which target version is supported?
- Are there removed or deprecated APIs?
- Are add-ons compatible?
- Are CRDs and webhooks compatible?
- Are PodDisruptionBudgets in place for critical workloads?
- Is etcd backed up?
- Is there a rollback or recovery plan?
Production upgrades are not merely a kubeadm upgrade exercise. An upgrade is a platform change process.
16.2 etcd and Cluster State
etcd stores cluster state. This includes API objects, Secrets, status information, and configurations.
Operational requirements:
- regular snapshots;
- secure snapshot retention;
- restore testing;
- monitoring of latency, disk, quorum, and leader changes;
- access protection;
- encryption of sensitive data;
- careful upgrade order.
Common misconception: a backup of YAML files does not replace an etcd backup. Conversely, an etcd backup does not automatically replace backups of external databases or volumes.
16.3 Node Lifecycle
Nodes are replaceable capacity, but they are not meaningless. Node state affects scheduling, performance, and availability.
Important concepts:
Ready/NotReady;- Node conditions such as MemoryPressure, DiskPressure, PIDPressure;
- cordon and drain;
- kubelet status;
- container runtime;
- CNI status;
- local images and cache;
- local volumes;
- kernel and cgroup configuration.
During maintenance, you typically mark a Node unschedulable first and then drain workloads in a controlled way. DaemonSets, PDBs, and local data must be considered.
16.4 Add-on Lifecycle
Add-ons are part of the platform and need lifecycle management.
For every add-on, you should know:
- Who owns it?
- Which version is running?
- How is it installed?
- How is it upgraded?
- Which CRDs does it provide?
- Which RBAC permissions does it have?
- Which webhooks does it register?
- What happens if it fails?
- How is it observed?
- How is it uninstalled?
Add-ons that sit in the API path are especially critical, for example admission webhooks. A faulty webhook can block cluster changes.
16.5 Backup and Restore Thinking
You need several backup layers:
| Layer | Backs up | Not sufficient for |
|---|---|---|
| Git | desired manifests | runtime status, Secrets, dynamic resources. |
| etcd snapshot | Kubernetes API state | external databases, volume contents. |
| Volume snapshot | block/filesystem state | application consistency without coordination. |
| Database backup | logical or physical data | Kubernetes resources. |
| Registry backup | images | cluster state and data. |
Restore order must be planned. Example: first cluster and CRDs, then operators, then Secrets, then PVCs or data restore, then workloads.
17. Extensibility
Kubernetes is extensible. This is one of the main reasons for its adoption, but also a source of complexity.
17.1 CRDs
CustomResourceDefinitions extend the Kubernetes API with custom resource types.
Example CRDs:
Certificatefrom cert-manager;Applicationfrom Argo CD;Prometheusfrom the Prometheus Operator;GatewayandHTTPRoutefrom Gateway API;- database resources from an operator.
CRDs define schema, versions, and validation for custom objects. A CRD alone, however, does not contain logic. The logic comes from controllers.
Important CRD questions:
- Which versions are supported?
- Are conversion webhooks used?
- Are status fields well defined?
- What happens on deletion?
- Are finalizers used?
- How are upgrades performed?
- Is the schema restrictive enough?
17.2 Controllers and Operators
An operator is a controller with domain-specific operational knowledge. It manages complex applications or external resources through Kubernetes APIs.
Examples:
- a database operator creates clusters, backups, and failover;
- cert-manager creates and renews certificates;
- External Secrets Operator synchronizes Secrets from external systems;
- Prometheus Operator manages monitoring resources;
- cloud operators manage external infrastructure.
Common misconception: operators reduce manual work, but they move complexity into controller logic. You still need to understand upgrades, backups, permissions, failure cases, and data consistency.
17.3 Webhooks and Admission Policies
Admission webhooks participate in the API write path.
| Type | Effect |
|---|---|
| Mutating Webhook | modifies objects before they are stored. |
| Validating Webhook | accepts or rejects objects. |
| ValidatingAdmissionPolicy | validates with declarative rules, without an external webhook service. |
Risks:
- an unreachable webhook blocks API changes depending on failure policy;
- a slow webhook increases API latency;
- faulty mutation creates changes that are hard to see;
- overly broad rules affect unexpected Namespaces;
- upgrading the webhook service can disrupt cluster operations.
Good rules:
- set a narrow scope;
- choose failure policy deliberately;
- keep timeouts low;
- run webhooks highly available;
- monitor webhook metrics and logs;
- document a break-glass process.
17.4 API Aggregation
API aggregation allows additional API servers behind the Kubernetes API Server. This mechanism is used for some extensions, for example aggregated APIs.
For the workshop, this distinction is usually enough:
- CRDs extend the API through Custom Resources in the Kubernetes API Server.
- API aggregation connects additional API servers.
- Admission webhooks validate or mutate objects during writes.
- Controllers react to stored objects and execute logic.
18. Decision Models
Kubernetes concepts become practical when you make decisions. The following models help in workshops and operations.
18.1 Deployment or StatefulSet?
| Question | If yes | If no |
|---|---|---|
| Does each Pod need a stable identity? | Evaluate StatefulSet. | Deployment is often enough. |
| Does each Pod need its own durable volume? | Evaluate StatefulSet. | Deployment with external state is often enough. |
| Are Pods interchangeable? | Deployment. | StatefulSet or another model. |
| Can traffic go to any replica? | Deployment with Service. | StatefulSet with headless Service or app-specific routing. |
18.2 ConfigMap or Secret?
| Question | Object |
|---|---|
| Does the value contain a password, token, certificate, or private key? | Secret or external secret system. |
| Can the value safely appear in logs or Git? | ConfigMap may be suitable. |
| Must the value be rotated and audited? | Secret management process. |
| Is the value only runtime configuration without secrecy? | ConfigMap. |
18.3 Service, Ingress, or Gateway?
| Need | Suitable concept |
|---|---|
| internal stable reachability | Service ClusterIP. |
| simple external TCP/UDP access through provider load balancer | Service LoadBalancer. |
| HTTP/HTTPS routing with the classic model | Ingress plus Ingress controller. |
| role-oriented, expressive routing | Gateway API plus Gateway controller. |
| direct Pod DNS names for StatefulSets | Headless Service. |
18.4 Setting Requests and Limits
| Situation | Decision |
|---|---|
| Workload is production-facing | Set CPU and memory requests. |
| Memory consumption must be hard-limited | Set a memory limit. |
| CPU latency is critical | Use CPU limits carefully or avoid them; perform load tests. |
| Namespace should constrain teams | Set ResourceQuota and LimitRange. |
| Autoscaling should work | Define requests and metrics cleanly. |
18.5 NetworkPolicy Strategy
| Maturity level | Strategy |
|---|---|
| Learning environment | Understand individual allow examples. |
| Team Namespace | Default deny plus explicit allows. |
| Production | Namespace strategy, DNS exceptions, egress control, tests, monitoring. |
| Regulated environment | Policy-as-code, review, audit, and exception process. |
19. Common Failure Patterns as a Concept Map
| Failure pattern | Likely concept area | First hypothesis |
|---|---|---|
Forbidden |
RBAC | Identity is not allowed to perform the action. |
Unauthorized |
Authentication | Token, certificate, or context is invalid. |
Pending |
Scheduling, storage | Node does not fit or PVC is not bound. |
ImagePullBackOff |
Runtime, registry, Secret | Image is unreachable or permission is missing. |
CrashLoopBackOff |
Application, config, probes | Process exits repeatedly. |
OOMKilled |
Resources | Memory limit exceeded. |
| Service without Endpoints | Labels, selectors, readiness | Service finds no ready Pods. |
| DNS does not work | CoreDNS, NetworkPolicy, CNI | DNS Pods or DNS traffic are disrupted. |
| Ingress returns 404 | Routing | Host, path, or IngressClass does not match. |
| Gateway Route does not attach | Gateway API | Route may not attach to listener or host does not match. |
PVC remains Pending |
Storage | StorageClass, provisioner, or zone does not fit. |
Pod remains Terminating |
Finalizer, Node, Volume | Cleanup logic or Node communication is stuck. |
| Rollout is stuck | Deployment, probes, resources | New Pods do not become ready. |
| HPA does not scale | Metrics, requests | Metrics are missing or requests are unsuitable. |
Node NotReady |
kubelet, runtime, network | Node does not reliably report to API Server. |
20. Workshop Exercises
These exercises are formulated conceptually. You will find concrete implementation steps in the command reference or lab files.
Exercise 1: Read Object Structure
Take a Deployment manifest and mark:
apiVersion;kind;metadata.name;metadata.namespace;- labels;
- selector;
- Pod template;
- container image;
- ports;
- probes;
- resources.
Then explain which fields express intent and which fields Kubernetes later adds as status.
Exercise 2: Understand the Selector Chain
Build this chain conceptually:
Deployment selector -> Pod labels -> Service selector -> EndpointSlice -> Traffic
Answer:
- What happens if the Pod label does not match the Deployment selector?
- What happens if the Pod label does not match the Service selector?
- What happens if the readiness probe fails?
Exercise 3: Classify Pod Failures
Map the following symptoms to conceptual areas:
ErrImagePull;FailedScheduling;Readiness probe failed;MountVolume.SetUp failed;Forbidden;OOMKilled.
Goal: do not immediately run commands from memory. First recognize the affected layer.
Exercise 4: Deployment vs StatefulSet
Evaluate three applications:
- HTTP API without local state;
- PostgreSQL database;
- log agent on every Node.
For each application, decide which workload resource fits conceptually and why.
Exercise 5: Draw the Network Model
Draw this flow for a web application:
Client -> Load Balancer -> Ingress/Gateway -> Service -> EndpointSlice -> Pod -> Container port
Mark where DNS, TLS, labels, readiness, and NetworkPolicies take effect.
Exercise 6: Plan Resources
The following observations are reported for an application:
- average memory usage of 80 MiB;
- peaks up to 180 MiB;
- occasional CPU spikes;
- latency is important;
- three replicas should run.
Derive:
- memory request;
- memory limit;
- CPU request;
- whether a CPU limit is useful;
- which metrics you would evaluate for HPA.
Exercise 7: Build Security Layers
Design for a Namespace:
- dedicated ServiceAccount;
- minimal RBAC role;
- Pod Security
restricted; - default-deny NetworkPolicy;
- allowed traffic from frontend to API;
- Secret-management rule.
Goal: recognize that security is not one YAML object.
Exercise 8: Make a Storage Decision
An application needs persistent storage. Answer:
- Does each Pod need its own volume?
- May the Pod move to another zone?
- Must the volume be read or written by multiple Pods?
- Are snapshots available?
- How is restore tested?
- Which data may be lost?
Exercise 9: Evaluate a CRD and Operator
Choose an operator, for example cert-manager or Prometheus Operator. Check:
- which CRDs are installed;
- which controllers run;
- which RBAC permissions are granted;
- which webhooks are used;
- what happens if the operator fails;
- how upgrades are documented.
Exercise 10: Reconstruct a Failure Chain
Symptom: after a rollout, the application is unreachable.
Work through the chain conceptually from top to bottom:
- Is the Deployment available?
- Are new Pods ready?
- Does the Service have Endpoints?
- Does Ingress or Gateway route to the correct Service?
- Are NetworkPolicies involved?
- Does DNS work?
- Does the application respond locally inside the Pod?
21. Glossary
| Term | Explanation |
|---|---|
| Admission Control | Validation or mutation of an API request before storage. |
| Affinity | Rule that influences scheduling onto specific Nodes or near other Pods. |
| Annotation | Non-selecting metadata on objects. |
| API Server | Central Kubernetes API and front door of the Control Plane. |
| CNI | Interface and plugin model for Pod networking. |
| ConfigMap | Object for non-secret configuration. |
| Container Runtime | Software that runs containers, for example containerd or CRI-O. |
| Controller | Control loop that observes and reconciles state. |
| CRD | CustomResourceDefinition; extension of the Kubernetes API. |
| CSI | Interface for storage integrations. |
| DaemonSet | Workload that runs one Pod on all or selected Nodes. |
| Deployment | Workload for stateless, interchangeable Pods with rollout functionality. |
| Desired State | Desired state described in spec. |
| EndpointSlice | Object containing backends of a Service. |
| etcd | Consistent key-value store for Kubernetes API data. |
| Event | Time-based hint about a decision, error, or state change. |
| Finalizer | Mechanism that delays deletion until cleanup work is complete. |
| Gateway API | Extensible API family for network access and routing. |
| HPA | HorizontalPodAutoscaler; scales Pod count based on metrics. |
| Ingress | Kubernetes resource for HTTP/HTTPS routing through an Ingress controller. |
| Job | Workload for tasks that run to completion. |
| kubelet | Node agent that runs Pods through the container runtime. |
| Label | Selectable key-value metadata. |
| Namespace | Scope for namespaced resources. |
| NetworkPolicy | Rule set for allowed Pod traffic at Layer 3/4. |
| Node | Machine or VM on which Pods run. |
| Operator | Controller with domain-specific operational knowledge. |
| Pod | Smallest deployable unit in Kubernetes. |
| PDB | PodDisruptionBudget; protects against too many voluntary disruptions. |
| PVC | PersistentVolumeClaim; storage request by a workload. |
| PV | PersistentVolume; provisioned storage in the cluster. |
| RBAC | Role-based authorization in Kubernetes. |
| Reconciliation | Continuous alignment between desired and actual state. |
| ReplicaSet | Maintains a stable number of identical Pods. |
| Secret | Object for sensitive data. |
| Service | Stable network abstraction for Pods. |
| ServiceAccount | Identity for workloads in the cluster. |
| StatefulSet | Workload for Pods with stable identity and often stable volume assignment. |
| Taint | Marking on Nodes that repels Pods without matching toleration. |
| Toleration | Permission for a Pod to be scheduled onto Nodes with a matching taint. |
| Volume | Storage mounted into a Pod. |
| Webhook | External service that can validate or mutate in the API path. |
22. Sources and Further Documentation
The following official documentation pages are useful for deeper study:
- Kubernetes Overview: https://kubernetes.io/docs/concepts/overview/
- Kubernetes Components: https://kubernetes.io/docs/concepts/overview/components/
- Cluster Architecture: https://kubernetes.io/docs/concepts/architecture/
- Kubernetes API Concepts: https://kubernetes.io/docs/reference/using-api/api-concepts/
- Objects in Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/
- Labels and Selectors: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- Namespaces: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
- Pods: https://kubernetes.io/docs/concepts/workloads/pods/
- Deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
- StatefulSets: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
- DaemonSets: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
- Jobs: https://kubernetes.io/docs/concepts/workloads/controllers/job/
- CronJobs: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
- Services: https://kubernetes.io/docs/concepts/services-networking/service/
- Gateway API: https://kubernetes.io/docs/concepts/services-networking/gateway/
- Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
- Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
- Resource Management for Pods and Containers: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
- RBAC Authorization: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
- Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/