1. Objective

This document describes the conceptual and operational architecture of Kubernetes for a practice-oriented workshop. It does not merely define terms; it connects core concepts with operational decisions, common failure patterns, and concrete manifests.

After the workshop, you should no longer see Kubernetes as a collection of isolated YAML files, but as a state-based control system: you describe a target state, Kubernetes validates it, stores it in the API, and lets controllers continuously work to align the actual state with that target state.

This document serves as:

• A reference for workshop modules and exercises.
• A basis for discussions about platform architecture.
• An entry point into troubleshooting and operational responsibility.
• Decision support for workload, network, storage, and security design.

This document is not intended to be a complete installation guide for a specific cloud provider or a particular distribution such as AKS, EKS, GKE, OpenShift, Rancher, or k3s. The architectural principles apply across clusters; individual implementations may differ in detail.

2. Learning Objectives

After working through this material, you will be able to:

• explain what the Control Plane, Worker Nodes, and add-ons are responsible for,
• trace the path from a manifest to a running Pod,
• classify Deployments, ReplicaSets, Services, ConfigMaps, Secrets, PersistentVolumeClaims, and Ingress/Gateway resources from an architectural perspective,
• decide when to use which Kubernetes object,
• systematically investigate common failure states such as ImagePullBackOff, CrashLoopBackOff, Pending, failed readiness checks, or network problems,
• set resource requests, limits, probes, and rollout strategies in a meaningful way,
• place security mechanisms such as RBAC, ServiceAccounts, Admission Control, Pod Security, and NetworkPolicies in their architectural context,
• document operational decisions instead of hiding them implicitly across scattered YAML files.

3. Architecture Overview

3.1. Core Idea

Kubernetes is a platform for running containerized applications. You describe applications, configuration, network access, persistence, and permissions through API objects. Kubernetes accepts these objects and coordinates their execution on a cluster of machines.

The important point is this: Kubernetes does not simply start containers. Kubernetes manages workloads. A workload consists of desired state, scheduling decisions, container runtime behavior, network identity, configuration, security, health checks, and status information.

flowchart LR
  user[You: manifest, CLI, or GitOps] --> api[Kubernetes API Server]
  api --> store[(etcd: cluster state)]
  api --> controllers[Controllers]
  controllers --> scheduler[Scheduler]
  scheduler --> nodes[Worker Nodes]
  nodes --> pods[Pods and containers]
  pods --> services[Services, DNS, Ingress/Gateway]
  pods --> obs[Logs, events, metrics]
  obs --> user

Kubernetes separates several areas of responsibility:

3.2. Control Plane, Worker Nodes, and Add-ons

A Kubernetes cluster consists of a Control Plane and one or more Worker Nodes.

The Control Plane maintains cluster state, validates API requests, and makes central control decisions. Worker Nodes run Pods. Add-ons provide functionality that Kubernetes needs as a platform, but that is not necessarily part of a monolithic core.

flowchart TB
  subgraph CP[Control Plane]
    api[kube-apiserver]
    etcd[(etcd)]
    sched[kube-scheduler]
    cm[kube-controller-manager]
    ccm[cloud-controller-manager]
  end

  subgraph W1[Worker Node]
    kubelet1[kubelet]
    proxy1[kube-proxy or eBPF dataplane]
    runtime1[Container Runtime]
    pod1[Pods]
  end

  subgraph ADD[Add-ons]
    dns[CoreDNS]
    ingress[Ingress/Gateway Controller]
    cni[CNI Plugin]
    csi[CSI Driver]
    metrics[Metrics/Logging]
  end

  api <--> etcd
  cm --> api
  sched --> api
  kubelet1 <--> api
  kubelet1 --> runtime1
  runtime1 --> pod1
  pod1 --> dns
  pod1 --> cni
  pod1 --> csi
  ingress --> api

3.3. Reconciliation Loop

The central operating mechanism in Kubernetes is the reconciliation loop. Controllers watch the API, compare desired state with actual state, and perform corrective actions.

Example: you define a Deployment with replicas: 3. If only two Pods are running, the responsible controller creates another Pod. If four Pods are running, one Pod is removed. Kubernetes is therefore not primarily command-oriented; it is state-oriented.

flowchart LR
  desired[Desired state\nmanifest in the API] --> compare[Controller compares]
  actual[Actual state\nPods, Nodes, network, storage] --> compare
  compare --> action[Corrective action]
  action --> actual

For operations, this means:

• You do not only investigate what you executed, but also which state is stored in the API.
• Many problems are caused not by a single command, but by conflicting target states.
• Controllers create derived objects. A Deployment creates ReplicaSets; ReplicaSets create Pods.
• Manual changes to derived objects are usually not persistent, because the parent controller overwrites them again.

4. Kubernetes Mental Model

4.1. Desired State

The desired state is the intended target state of a resource. You describe it in YAML or JSON, usually in YAML. Kubernetes stores this state in etcd and tries to bring the actual state into alignment with it.

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27

With this, you are not saying: “Start three containers now.” You are saying: “Continuously ensure that three matching Pod replicas exist.” That is a fundamental difference.

4.2. Declarative API

Kubernetes uses a declarative API. You describe the outcome, not every intermediate step. This fits GitOps and Infrastructure as Code well, because the desired state becomes versionable and reviewable.

Typical workflow:

kubectl apply -f deployment.yaml
kubectl get deployment web
kubectl rollout status deployment/web

Important consequences:

• kubectl apply is designed for repeatedly applying manifests.
• kubectl create is more suitable for one-time creation.
• kubectl edit is convenient in a lab, but risky in production because changes are not automatically versioned.
• GitOps controllers such as Argo CD or Flux expect Git to be the authoritative desired state.

4.3. Resources, Objects, and Controllers

A resource is a type in the Kubernetes API, for example Pod, Deployment, or Service. An object is a concrete instance of that resource, for example deployment/web.

Almost every object contains:

You normally change spec, not status. The status is the result of controller work and shows what Kubernetes observes.

4.4. Names, Labels, Selectors, and Annotations

Names identify objects within their scope. A Deployment name must be unique within a Namespace.

Labels are structured key-value pairs used to select and group objects. Services, ReplicaSets, NetworkPolicies, and many other resources use label selectors.

Annotations store additional metadata that is not primarily intended for selection. Many controllers use annotations for configuration, integrations, or tooling hints.

Example:

metadata:
  name: web
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/component: frontend
    app.kubernetes.io/part-of: shop
    environment: workshop
  annotations:
    owner: platform-team
    documentation: "https://example.invalid/runbooks/web"

Workshop recommendation:

• Use a consistent label schema.
• Use labels for selection and grouping.
• Use annotations for descriptive or tool-specific additional information.
• Plan selectors before creating resources. For Deployments, the selector is effectively a core part of the resource identity and should not be changed later.

4.5. Namespaces

Namespaces logically divide a cluster into areas. You use them to organize resources, permissions, quotas, NetworkPolicies, and naming scopes. However, a Namespace is not a complete security boundary in the same way as a separate cluster.

Typical Namespace strategies:

For production workloads, you should avoid the default Namespace. Create explicit Namespaces and document who owns them, which quotas apply, and which access is allowed.

Example:

apiVersion: v1
kind: Namespace
metadata:
  name: workshop
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

5. Components

5.1. Control Plane

5.2. Worker Node

5.3. Cluster Add-ons

6. Core Kubernetes Objects

7. From Manifest to Running Pod

When you apply a manifest, several steps happen. These steps are essential for troubleshooting.

sequenceDiagram
  participant You
  participant API as kube-apiserver
  participant Etcd as etcd
  participant Ctrl as Controller
  participant Sched as Scheduler
  participant Kubelet as kubelet
  participant Runtime as Container Runtime

  You->>API: kubectl apply -f deployment.yaml
  API->>API: Authentication, authorization, admission, validation
  API->>Etcd: Store object
  Ctrl->>API: Watch Deployment
  Ctrl->>API: Create ReplicaSet and Pods
  Sched->>API: Watch unscheduled Pods
  Sched->>API: Set Node binding
  Kubelet->>API: Fetch PodSpec for its own Node
  Kubelet->>Runtime: Pull images, start containers
  Runtime-->>Kubelet: Status and logs
  Kubelet->>API: Update PodStatus

Important diagnostic points:

8. Workloads, Deployments, and Updates

8.1. Pods as the Smallest Unit

A Pod is the smallest unit Kubernetes can directly run. A Pod contains one or more containers that share a network namespace, IP address, and optionally volumes.

In practice, a Pod usually contains exactly one application container. You should only use multiple containers in a Pod when they are tightly coupled and must run together on the same Node.

Common multi-container patterns:

Pods are inherently short-lived. When a Pod is deleted, a replacement Pod receives a new UID and usually a new IP. For that reason, you should not address Pods directly; work through Services, labels, and controllers instead.

8.2. ReplicaSets and Deployments

A ReplicaSet ensures that a specific number of identical Pods is running. A Deployment manages ReplicaSets and provides additional capabilities such as rollouts, rollbacks, and rollout history.

In the workshop, you will almost always use Deployments for stateless applications, not ReplicaSets directly.

flowchart TB
  dep[Deployment] --> rs1[ReplicaSet Revision 1]
  dep --> rs2[ReplicaSet Revision 2]
  rs1 --> oldpods[old Pods]
  rs2 --> newpods[new Pods]

Basic rules:

• Change the Pod template of a Deployment when you want to trigger a new rollout.
• Scale through spec.replicas or an autoscaler.
• Do not edit Pods created by a Deployment as a permanent solution.
• Choose labels and selectors deliberately, because they define the relationship between Deployment, ReplicaSet, Pods, and Service.

8.3. Rolling Updates and Rollbacks

A rolling update gradually replaces old Pods with new Pods. This allows an application to remain available while a new version is rolled out.

Important parameters:

Useful commands:

kubectl rollout status deployment/web -n workshop
kubectl rollout history deployment/web -n workshop
kubectl rollout undo deployment/web -n workshop
kubectl describe deployment web -n workshop

If a rollout stalls, inspect the new Pods first:

kubectl get pods -n workshop -l app=web
kubectl describe pod <pod-name> -n workshop
kubectl logs <pod-name> -n workshop

8.4. Example: Deployment with Resources and Probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: workshop
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/component: frontend
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: web
      app.kubernetes.io/component: frontend
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
        app.kubernetes.io/component: frontend
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 80
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          startupProbe:
            httpGet:
              path: /
              port: http
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10

Assessment of this example:

• requests give the Scheduler a reliable basis for placement decisions.
• limits prevent unlimited resource usage, but CPU limits can also lead to throttling.
• startupProbe protects slow-starting applications from premature liveness restarts.
• readinessProbe controls whether the Pod should receive traffic.
• livenessProbe should only detect real deadlocks or conditions that cannot self-heal.

9. Network Architecture

9.1. Kubernetes Network Model

Kubernetes assumes a flat Pod network: every Pod can, in principle, reach every other Pod through its Pod IP without network address translation at the Pod level. How this model is implemented technically depends on the CNI plugin.

Important network layers:

9.2. Services and Service Discovery

Pods are short-lived and receive new IP addresses. Services provide a stable abstraction over a group of Pods.

Example:

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: workshop
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: web
    app.kubernetes.io/component: frontend
  ports:
    - name: http
      port: 80
      targetPort: http

Service types:

DNS names usually follow this pattern:

<service>.<namespace>.svc.cluster.local

For a Service named web in the Namespace workshop, that becomes:

web.workshop.svc.cluster.local

Within the same Namespace, web is often enough because the Pod's DNS search list is expanded automatically.

9.3. Ingress and Gateway API

Ingress describes HTTP and HTTPS routing from outside the cluster to Services inside the cluster. An Ingress object only has an effect if a matching Ingress Controller is installed.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: workshop
spec:
  ingressClassName: nginx
  rules:
    - host: web.example.invalid
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

For new platform architectures, you should also evaluate the Gateway API. This API is more role-oriented and expressive than classic Ingress, especially when multiple teams separately own infrastructure, routing, and applications.

Conceptual difference:

9.4. NetworkPolicies

Without NetworkPolicies, many clusters allow Pod-to-Pod communication by default. A NetworkPolicy defines which ingress and/or egress traffic is allowed for selected Pods. Enforcement depends on the CNI plugin.

Example: only Pods with the label app.kubernetes.io/name: web may reach a database on port 5432.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-db
  namespace: workshop
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: web
      ports:
        - protocol: TCP
          port: 5432

Operational rules:

• Check whether your CNI actually supports NetworkPolicies.
• Start with clear default-deny policies, but plan exceptions deliberately.
• Consider DNS egress when you introduce egress policies.
• Document allowed communication paths as part of the application architecture.

10. Storage and Persistence

Container filesystems are ephemeral. When a Pod is replaced, you must not assume that local data is retained. Kubernetes therefore separates workload and storage through Volumes, PersistentVolumes, and PersistentVolumeClaims.

Important terms:

Example PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: workshop
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

Typical architecture questions:

• Is the application stateless or stateful?
• Does storage need to be available across zones or regions?
• Which accessMode is required: ReadWriteOnce, ReadOnlyMany, ReadWriteMany, or ReadWriteOncePod?
• What happens when a PVC is deleted?
• How are backups, restores, and migrations tested?
• What latency and IOPS does the application actually need?

Running databases inside a Kubernetes cluster requires substantially more architectural work than running stateless web services. StatefulSets solve identity and ordered startup, but they do not replace a backup concept, a replication strategy, or a disaster recovery procedure.

11. Configuration and Secrets

Configuration should not be baked into the container image. Kubernetes separates application code, non-sensitive configuration, and sensitive data.

Example ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: workshop
data:
  LOG_LEVEL: "info"
  FEATURE_CHECKOUT: "enabled"

Example Secret:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: workshop
type: Opaque
stringData:
  username: app
  password: change-me-in-real-life

Use in a Pod:

envFrom:
  - configMapRef:
      name: web-config
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: database-credentials
        key: password

Operational guidance:

• Secrets are not automatically equivalent to strong encryption. Plan encryption at rest, RBAC, audit, and, where needed, external secret stores.
• ConfigMap and Secret updates behave differently depending on how they are mounted or injected. Environment variables are only refreshed on Pod restart; mounted files can be updated, but not immediately in every special case.
• Avoid exposing secrets in logs, events, command-line arguments, and CI output.
• Use separate Secrets per application and purpose, not one global catch-all Secret.

12. Resource Management and Scheduling

12.1. Requests, Limits, and QoS

Kubernetes schedules Pods based on their requests. Limits restrict maximum usage.

Example:

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 512Mi

QoS classes:

Workshop rule: always set requests for production-like examples. Without requests, the Scheduler cannot make realistic capacity decisions.

12.2. Node Selection, Affinity, and Taints

The Scheduler chooses Nodes based on many criteria. You can influence that choice.

Example toleration:

tolerations:
  - key: "workload"
    operator: "Equal"
    value: "platform"
    effect: "NoSchedule"

Example Node Affinity:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - eu-central-1a
                - eu-central-1b

12.3. Quotas and Limits per Namespace

ResourceQuotas limit the aggregate resource consumption of a Namespace. LimitRanges can set default values or minimum and maximum values for Pods and containers.

Example ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: workshop-quota
  namespace: workshop
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"

Architecture note: quotas do not protect against every form of overload, but they prevent a Namespace from consuming unlimited resources or creating unlimited objects.

12.4. Autoscaling and Disruption Budgets

The Horizontal Pod Autoscaler scales replicas based on observed metrics. For this, you need suitable metric sources and meaningful requests.

Example HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: workshop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

A PodDisruptionBudget limits voluntary disruptions, for example during Node maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: workshop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: web
      app.kubernetes.io/component: frontend

Important: a PDB does not protect against all failures. It mainly applies to voluntary disruptions such as kubectl drain, not to unplanned Node crashes.

13. Health Checks and Lifecycle

Kubernetes provides several types of probes. You should not treat them as interchangeable.

Simplified Pod lifecycle:

stateDiagram-v2
  [*] --> Pending
  Pending --> Running
  Running --> Succeeded
  Running --> Failed
  Running --> Terminating
  Terminating --> [*]

Operational rules:

• Readiness should check dependencies that are relevant for accepting traffic.
• Liveness should only check states that are likely to be fixed by a restart.
• Startup probes are useful for slow applications, JVMs, migrations, or large cache warmups.
• Probes must not overload the application themselves.
• Probe endpoints should be fast, deterministic, and ideally avoid expensive queries.

14. Security

14.1. Access to the Kubernetes API

The API Server is the central control point. Every request passes through several phases:

1. Authentication: who is making the request?
2. Authorization: is this identity allowed to perform the action?
3. Admission Control: is the object accepted, rejected, or modified?
4. Persistence: is the new state stored in etcd?

flowchart LR
  req[API request] --> authn[Authentication]
  authn --> authz[Authorization]
  authz --> adm[Admission Control]
  adm --> val[Validation]
  val --> etcd[(etcd)]

For local access, kubectl uses a kubeconfig. A kubeconfig contains cluster, user, and context information. Treat kubeconfig files as sensitive credentials.

14.2. RBAC and ServiceAccounts

RBAC defines which subjects may execute which verbs on which resources.

Example: a ServiceAccount may read Pods in the Namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-reader
  namespace: workshop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: workshop
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader
  namespace: workshop
subjects:
  - kind: ServiceAccount
    name: pod-reader
    namespace: workshop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

Operational rules:

• Grant minimum required permissions.
• Avoid wildcards such as * unless they are strictly necessary.
• Check permissions with kubectl auth can-i.
• Give workloads their own ServiceAccounts instead of using the default ServiceAccount without control.
• Be especially careful with permissions on secrets, nodes, pods/exec, pods/attach, roles, rolebindings, clusterroles, and clusterrolebindings.

14.3. Admission Control and Pod Security

Admission controllers check API requests after authentication and authorization, but before storage. They can validate or mutate objects.

Typical use cases:

• Enforcing labels and ownership metadata.
• Blocking privileged containers.
• Enforcing resource requests.
• Blocking images from unapproved registries.
• Setting default values.
• Checking Pod Security Standards.

Pod Security Admission can label Namespaces with security levels:

Example Namespace labels:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

14.4. Secret Protection

Secrets require both technical and organizational protection measures.

Checklist:

• Enable encryption at rest for Secrets in the cluster.
• Restrict get, list, and watch permissions on Secrets very tightly.
• Avoid Secret values in environment variables if process lists, dumps, or debugging could expose them.
• Prefer external secret stores when governance, rotation, and audit require them.
• Verify that backup systems store Secrets encrypted.
• Rotate Secrets and document rotation as an operational process.

14.5. Image and Supply Chain Security

Kubernetes runs the images you reference. The platform does not automatically eliminate all risks around image provenance and content.

Recommendations:

• Use trusted registries.
• Prefer immutable image references through digests instead of tags only.
• Scan images for known vulnerabilities.
• Sign images or verify signatures where your platform supports it.
• Avoid latest in production manifests.
• Set imagePullPolicy deliberately.
• Use slim base images and remove unnecessary tools.

15. Observability

Kubernetes provides several layers of observability. No single layer is sufficient on its own.

Minimal observability approach in the workshop:

kubectl get all -n workshop
kubectl get events -n workshop --sort-by=.lastTimestamp
kubectl describe deployment web -n workshop
kubectl describe pod <pod-name> -n workshop
kubectl logs <pod-name> -n workshop
kubectl top pods -n workshop

For production, you additionally need:

• centralized log storage,
• metrics with alerts,
• dashboards based on service-level indicators,
• audit logs,
• runbooks,
• clear ownership for each Namespace and application.

16. Troubleshooting

16.1. Systematic Approach

Use a fixed sequence so you do not troubleshoot randomly.

1. Check context

   bashCopy code

   kubectl config current-context
   kubectl config view --minify
   kubectl get ns
   

2. Check objects in the Namespace

   bashCopy code

   kubectl get all -n workshop
   kubectl get deploy,rs,pods,svc,endpointslice -n workshop
   

3. Read events

   bashCopy code

   kubectl get events -n workshop --sort-by=.lastTimestamp
   

4. Describe the affected object

   bashCopy code

   kubectl describe pod <pod-name> -n workshop
   

5. Check logs

   bashCopy code

   kubectl logs <pod-name> -n workshop
   kubectl logs <pod-name> -n workshop --previous
   

6. Understand the owner chain

   bashCopy code

   kubectl get pod <pod-name> -n workshop -o yaml | grep -A 10 ownerReferences
   

7. Test the network in isolation

   bashCopy code

   kubectl run debug -n workshop --rm -it --image=curlimages/curl -- sh
   

16.2. Common Failure Patterns

16.3. Useful kubectl Commands

# Overview
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get all -n workshop

# Status and events
kubectl describe pod <pod-name> -n workshop
kubectl get events -n workshop --sort-by=.lastTimestamp

# Logs
kubectl logs <pod-name> -n workshop
kubectl logs <pod-name> -n workshop --previous
kubectl logs deployment/web -n workshop

# Rollouts
kubectl rollout status deployment/web -n workshop
kubectl rollout history deployment/web -n workshop
kubectl rollout undo deployment/web -n workshop

# Labels and selectors
kubectl get pods -n workshop --show-labels
kubectl get pods -n workshop -l app.kubernetes.io/name=web

# Services and endpoints
kubectl get svc,endpointslice -n workshop
kubectl describe svc web -n workshop

# Show YAML
kubectl get deployment web -n workshop -o yaml
kubectl explain deployment.spec.template.spec.containers.resources

# Check permissions
kubectl auth can-i get pods -n workshop
kubectl auth can-i list secrets -n workshop --as=system:serviceaccount:workshop:pod-reader

# Start debug Pod
kubectl run debug -n workshop --rm -it --image=busybox:1.36 -- sh

17. Architecture Decisions

Document architecture decisions explicitly. Kubernetes manifest files show what is configured, but not always why it was configured that way.

Guiding questions:

Desired State and Ownership

• Where is the authoritative desired state: Git, CI/CD system, Helm repository, GitOps controller, or manual API changes?
• Who is allowed to change production manifests?
• How are changes reviewed, tested, and rolled out?
• How are rollbacks performed?

Namespace and Tenant Model

• Which Namespaces exist and who owns them?
• Are Namespaces split by team, application, environment, or tenant?
• Which quotas and LimitRanges apply?
• Which NetworkPolicies apply by default?

Workload Model

• Is the application stateless or stateful?
• Is a Deployment sufficient, or do you need a StatefulSet, DaemonSet, Job, or CronJob?
• What is the minimum required replica count?
• Which update strategy is acceptable?
• How is availability ensured during maintenance?

Resources

• Which requests and limits are realistic?
• How is load tested?
• Is there an HPA, VPA, or Cluster Autoscaler?
• How do you detect CPU throttling, OOMKills, and Node pressure?

Network

• Which Services must be reachable internally?
• Which external endpoints exist?
• Is Ingress or Gateway API used?
• Which TLS strategy applies?
• Which communication paths are allowed through NetworkPolicies?

Storage

• Which data must be persistent?
• Which StorageClass is used?
• Which backup and restore processes are tested?
• What happens during a zone or Node failure?
• Which data may be deleted and which data must not be deleted?

Security

• Which ServiceAccounts does the application use?
• Which RBAC permissions are actually required?
• Which admission policies apply?
• How are Secrets created, stored, rotated, and audited?
• Which images may be used?

Observability and Operations

• Which logs, metrics, and traces are mandatory?
• Which alerts indicate real user impact rather than only technical symptoms?
• Are runbooks available?
• Who responds to incidents?
• How does the team learn from incidents?

18. Workshop Exercises

The following exercises match this material. You can work through them in sequence or select individual modules.

Exercise 1: Create Namespace and Baseline

Goal: create a dedicated Namespace and verify your context.

kubectl create namespace workshop
kubectl config set-context --current --namespace=workshop
kubectl get ns

Discussion:

• Why is a dedicated Namespace better than default?
• Which labels would you set for ownership and Pod Security?

Exercise 2: Roll Out a Deployment

Goal: create a Deployment with three replicas and observe the controller chain.

kubectl apply -f deployment.yaml
kubectl get deploy,rs,pods -n workshop
kubectl rollout status deployment/web -n workshop

Discussion:

• Which objects did you create directly?
• Which objects did Kubernetes derive from them?
• What happens if you manually delete a Pod?

Exercise 3: Test Service Discovery

Goal: create a Service and test DNS from a debug Pod.

kubectl apply -f service.yaml
kubectl run debug --rm -it --image=curlimages/curl -- sh
curl http://web.workshop.svc.cluster.local

Discussion:

• Why do you address the Service rather than Pod IPs?
• How do you know whether a Service has endpoints?

Exercise 4: Intentionally Create a Failure

Goal: change the image to a non-existent tag and analyze the error.

kubectl set image deployment/web web=nginx:does-not-exist -n workshop
kubectl rollout status deployment/web -n workshop
kubectl get pods -n workshop
kubectl describe pod <pod-name> -n workshop

Discussion:

• Where is the actual error message shown?
• Why might the Deployment remain partially available?
• How do you perform a rollback?

Exercise 5: Understand Readiness and Liveness

Goal: deliberately set an incorrect readiness probe and observe Service endpoints.

kubectl get endpointslice -n workshop
kubectl describe pod <pod-name> -n workshop

Discussion:

• Why is the container running but not receiving traffic?
• What is the difference between readiness and liveness?

Exercise 6: Check RBAC

Goal: create a ServiceAccount with minimal permissions and verify access.

kubectl auth can-i list pods -n workshop --as=system:serviceaccount:workshop:pod-reader
kubectl auth can-i list secrets -n workshop --as=system:serviceaccount:workshop:pod-reader

Discussion:

• Why is list secrets especially sensitive?
• When do you need a Role, and when do you need a ClusterRole?

Exercise 7: Introduce NetworkPolicy

Goal: restrict communication and verify whether your CNI enforces policies.

kubectl apply -f networkpolicy.yaml
kubectl get networkpolicy -n workshop

Discussion:

• What happens in a CNI without NetworkPolicy support?
• Which egress exceptions do you need for DNS?

19. Scope and Boundaries

Kubernetes is not identical to the tools and platforms around Kubernetes. A clean architecture separates responsibilities.

20. Glossary

21. Official References

This material is based on the official Kubernetes documentation. For details and version-specific differences, check the documentation for the Kubernetes version you are using.

• Kubernetes Components: https://kubernetes.io/docs/concepts/overview/components/
• Pods: https://kubernetes.io/docs/concepts/workloads/pods/
• Pod Lifecycle: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
• Deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
• Services: https://kubernetes.io/docs/concepts/services-networking/service/
• DNS for Services and Pods: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
• Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
• Gateway API: https://kubernetes.io/docs/concepts/services-networking/gateway/
• Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• ConfigMaps: https://kubernetes.io/docs/concepts/configuration/configmap/
• Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
• Resource Management for Pods and Containers: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
• Labels and Selectors: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
• Namespaces: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
• RBAC Good Practices: https://kubernetes.io/docs/concepts/security/rbac-good-practices/
• Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/
• Good practices for Kubernetes Secrets: https://kubernetes.io/docs/concepts/security/secrets-good-practices/
• Horizontal Pod Autoscaling: https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/
• Pod Disruption Budgets: https://kubernetes.io/docs/tasks/run-application/configure-pdb/