1. Purpose

This document collects assessed best practices for Kubernetes. It is intended for workshops, reviews, and final projects. The goal is not merely to write YAML, but to make decisions traceable, identify risks early, and operate changes safely.

The rules are intentionally practical. Each best practice answers at least one of these questions:

• Why is this rule important?
• How do you implement it concretely?
• How do you verify that it is being followed?
• Which typical failure modes appear when you ignore it?

What you should be able to do after the workshop

After working through this document, you should be able to:

• write Kubernetes manifests in a structured way,
• roll out and roll back Deployments safely,
• classify Services, Ingress, Gateway API, and NetworkPolicies correctly,
• define resource requirements and scaling behavior in a traceable way,
• use RBAC, ServiceAccounts, Secrets, and Pod Security Standards correctly from an architectural perspective,
• analyze common failure modes systematically with kubectl,
• apply review checklists to real projects.

Scope

These best practices apply regardless of whether you run Kubernetes locally, in your own data center, or with a cloud provider. Environment-specific details such as LoadBalancer implementation, CSI driver, CNI plugin, IAM integration, managed control plane, or registry integration must be added for the specific platform you use.

2. Guiding Principles

Kubernetes becomes more reliable when you follow a small set of core principles.

Rule of thumb

A Kubernetes solution is not good merely because a Pod is running. It is good when you can explain, reproduce, secure, observe, and roll back why it is running.

3. Manifest and Repository Standards

3.1. Work Declaratively

Write resources as manifests and manage them under version control. Avoid manual changes in the cluster that cannot be traced back to the repository.

Best practice

• Store all production-relevant resources as YAML, Helm charts, Kustomize overlays, or GitOps configuration.
• Use kubectl apply or a GitOps tool; do not rely permanently on kubectl edit.
• Check changes in advance with kubectl diff and --dry-run=server.
• Document which source is authoritative: Git, Helm release, GitOps controller, or another deployment system.

Good validation

kubectl diff -f k8s/
kubectl apply --dry-run=server -f k8s/
kubectl apply -f k8s/

Poor pattern

kubectl edit deployment web
kubectl set image deployment/web web=registry.example.com/web:latest

This may be legitimate during an incident. Afterwards, however, you must bring the change back into the declarative source. Otherwise, configuration drift is created.

3.2. Use Traceable Resource Names

Names should be short, unique, and meaningful. Avoid project-internal slang, arbitrary abbreviations, and names that do not make the environment or purpose understandable.

Recommended pattern

<app>-<component>
<app>-<component>-<purpose>

Examples

Rule

• The name describes the functional purpose and the technical role.
• Environment-specific information usually belongs in the namespace, overlay, or labels, not necessarily in every resource name.
• A name should remain understandable in logs, events, and monitoring.

3.3. Use Labels and Annotations Consistently

Labels are for selection, grouping, and automation. Annotations are for additional metadata that is not primarily used for selection.

Recommended standard labels

metadata:
  labels:
    app.kubernetes.io/name: shop-api
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: shop
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/version: "1.4.2"
    environment: dev
    team: platform

Important

• Service selectors must match Pod labels.
• Labels that drive selectors must not be changed casually.
• Use labels for monitoring, cost allocation, ownership, and policy exceptions.
• Use annotations for build information, Git revisions, documentation links, or tool-specific hints.

Validation

kubectl get pods --show-labels
kubectl get svc shop-api -o yaml
kubectl get endpointslice -l kubernetes.io/service-name=shop-api

3.4. Separate Base, Environments, and Secrets

A clean structure prevents environment details from being mixed together and reduces the risk of accidentally committing secrets.

Example structure

k8s/
  base/
    deployment.yaml
    service.yaml
    serviceaccount.yaml
    networkpolicy.yaml
  overlays/
    dev/
      kustomization.yaml
      patch-resources.yaml
      patch-ingress.yaml
    test/
      kustomization.yaml
      patch-resources.yaml
    prod/
      kustomization.yaml
      patch-resources.yaml
      patch-pdb.yaml

Best practice

• Shared resources live in base.
• Environment-specific differences live in overlays.
• Secrets are not stored in plaintext in the repository.
• Provider-specific resources are documented separately.
• Example values are clearly identifiable as examples.

Review question

Can you determine from the repository what differs between dev, test, and prod without querying the cluster?

3.5. Validate Changes Before Applying Them

Kubernetes may accept syntactically valid YAML that is still incorrect from an operational or architectural perspective. That is why you need multiple validation layers.

Validation layers

Minimum workshop standard

kubectl apply --dry-run=server -f manifest.yaml
kubectl diff -f manifest.yaml
kubectl apply -f manifest.yaml
kubectl rollout status deployment/<name>

4. Workload Best Practices

4.1. Use the Right Workload Type

Not every workload belongs in a Deployment.

Rule

• Use Deployments for standard stateless workloads.
• Use StatefulSets only when you need stable identity, ordered startup, or persistent volumes per replica.
• Use Jobs and CronJobs for finite tasks.
• Use DaemonSets only when one instance truly must run on every Node or on a selected set of Nodes.

4.2. Make Deployments Rollback-Capable

A Deployment is only cleanly designed when rollout and rollback can be controlled.

Recommended Deployment example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
  labels:
    app.kubernetes.io/name: shop-api
    app.kubernetes.io/component: api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: shop-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: shop-api
        app.kubernetes.io/component: api
    spec:
      serviceAccountName: shop-api
      automountServiceAccountToken: false
      containers:
        - name: shop-api
          image: registry.example.com/shop-api:1.4.2
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 20
            periodSeconds: 20
            timeoutSeconds: 2
            failureThreshold: 3
          startupProbe:
            httpGet:
              path: /startup
              port: http
            periodSeconds: 5
            failureThreshold: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 10001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

Explanation

• replicas: 3 protects against the loss of individual Pods.
• maxUnavailable: 0 prevents Kubernetes from intentionally reducing availability during a rollout.
• maxSurge: 1 allows one additional instance during the rollout.
• revisionHistoryLimit keeps old ReplicaSets available for rollbacks.
• readinessProbe determines whether traffic should be routed to the Pod.
• livenessProbe determines whether Kubernetes should restart a container.
• startupProbe protects slow-starting applications from premature liveness checks.
• securityContext reduces container privileges.

Validation

kubectl rollout status deployment/shop-api
kubectl rollout history deployment/shop-api
kubectl describe deployment shop-api
kubectl get rs -l app.kubernetes.io/name=shop-api

4.3. Plan for Graceful Shutdown

Kubernetes does not terminate Pods immediately. It sends a termination signal and waits for the configured grace period. Your application must use that time to finish requests, close connections, and stop accepting new work.

Best practice

• Implement signal handling for SIGTERM.
• Configure the readinessProbe so the Pod is removed from traffic before termination.
• Choose terminationGracePeriodSeconds according to the application’s behavior.
• Use preStop only when you truly need an explicit delay or deregistration action.
• Check whether your Load Balancer or Ingress Controller has its own drain timing.

Example

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: shop-api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

Risk

A shutdown window that is too short causes interrupted requests. A shutdown window that is too long slows down rollouts, node drains, and scaling operations.

4.4. Use Probes Deliberately

Probes do not replace good application design. They provide the signals Kubernetes uses to make decisions.

Best practice

• Use a readinessProbe for almost all services.
• Use a livenessProbe only when a restart can actually recover a stuck state.
• Use a startupProbe for slow starts.
• Let readiness check relevant dependencies, but do not make it overly aggressive.
• Liveness should not hard-depend on external systems. Otherwise, a database problem can trigger a restart storm.

Anti-pattern

livenessProbe:
  httpGet:
    path: /deep-check-with-database-and-third-party-api
    port: 8080

If the database is briefly unavailable, Kubernetes restarts all Pods. That makes the incident worse.

4.5. Distribute Pods Intentionally

Multiple replicas only help if they do not all end up on the same Node or in the same failure zone.

Options

Example: topology spread

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: shop-api

Rule

• For production services with multiple replicas, actively verify distribution.
• For critical workloads, also consider zone-level distribution.
• Use hard scheduling rules only when you have enough capacity. Otherwise, Pods remain Pending.

4.6. Protect Availability with PodDisruptionBudgets

A PodDisruptionBudget limits voluntary disruptions, such as node drains, cluster maintenance, or controlled evictions.

Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shop-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: shop-api

Best practice

• Set PDBs for production workloads with multiple replicas.
• Do not use PDBs as a substitute for a sufficient replica count.
• Check whether PDBs can block maintenance work.
• Be especially careful with quorum-based systems.

Validation

kubectl get pdb
kubectl describe pdb shop-api
kubectl drain <node> --ignore-daemonsets --dry-run=server

5. Resource Management and Scaling

5.1. Set Resource Requests

Requests provide scheduling information to the Scheduler. Without requests, Kubernetes has less reliable placement, quotas become harder to reason about, autoscaling becomes unreliable, and Nodes may be overcommitted.

Best practice

• Set resources.requests.cpu and resources.requests.memory for every container.
• Start with measured values, not wishful values.
• Use monitoring data, load tests, and operational experience.
• Document assumptions: expected requests per second, memory profile, startup behavior, peak load.

Example

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi

Note on CPU

CPU requests influence scheduling and CPU allocation. CPU limits can lead to throttling. Set CPU limits deliberately and observe actual behavior.

Note on memory

Memory limits are hard boundaries. If a container exceeds its limit, it can be terminated with OOMKilled. That is why you need realistic values and alerts.

5.2. Set Limits Deliberately

Limits protect Nodes and neighboring workloads, but they can also create failures.

Best practice

• Set at least CPU and memory requests.
• Set memory limits for almost all application containers.
• Validate CPU limits with latency and throughput tests.
• Limit temporary storage when applications write significant data.
• Use dashboards for CPU throttling, working set memory, and OOM events.

5.3. Use Namespaces, ResourceQuota, and LimitRange

Namespaces are not a full security boundary, but they are an important organizational and control layer.

Namespace example

apiVersion: v1
kind: Namespace
metadata:
  name: shop-dev
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

ResourceQuota example

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: shop-dev
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.memory: 16Gi
    pods: "30"

LimitRange example

apiVersion: v1
kind: LimitRange
metadata:
  name: default-container-limits
  namespace: shop-dev
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 256Mi

Best practice

• Use namespaces per team, application, environment, or tenant.
• Set ResourceQuotas for shared clusters.
• Set LimitRanges so missing requests do not go unnoticed.
• Document which resources a team can expect to use.

5.4. Scale Based on Metrics

Scaling without metrics is guesswork. Kubernetes can scale horizontally when suitable metrics are available.

HorizontalPodAutoscaler example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shop-api
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Best practice

• Use HPA for stateless workloads with variable load.
• Ensure that Metrics Server or appropriate custom metrics are available.
• Set requests correctly; otherwise, CPU utilization targets are not meaningful.
• Define minReplicas and maxReplicas based on operational requirements.
• Validate scaling behavior with load tests.
• Combine workload autoscaling with node autoscaling when needed.

Validation

kubectl get hpa
kubectl describe hpa shop-api
kubectl top pods
kubectl top nodes

6. Networking and Service Discovery

6.1. Use Services as the Stable Access Layer

Pods are ephemeral. Services are the stable access layer inside the cluster.

Service example

apiVersion: v1
kind: Service
metadata:
  name: shop-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: shop-api
  ports:
    - name: http
      port: 80
      targetPort: http

Best practice

• Use ClusterIP for internal communication.
• Use LoadBalancer only when a service must be directly reachable from outside.
• Use Ingress or Gateway API for HTTP/HTTPS routing.
• Name ports so probes, Services, and policies remain easier to read.
• Always check whether the Service has Endpoints or EndpointSlices.

Validation

kubectl get svc shop-api
kubectl get endpoints shop-api
kubectl get endpointslice -l kubernetes.io/service-name=shop-api

6.2. Understand DNS Inside the Cluster

Kubernetes provides DNS names for Services. Typically, a Service in the same namespace can be reached by its short name. Across namespaces, use the fully qualified name.

Examples

shop-api
shop-api.shop-dev
shop-api.shop-dev.svc.cluster.local

Best practice

• Use Service names instead of Pod IPs.
• Use fully qualified DNS names when communicating across namespaces.
• Debug DNS problems from a debug Pod.
• Distinguish DNS failures from NetworkPolicy or Service selector problems.

Validation

kubectl run dns-debug --rm -it --image=busybox:1.36 --restart=Never -- nslookup shop-api.shop-dev.svc.cluster.local
kubectl get pods -n kube-system -l k8s-app=kube-dns

6.3. Use Ingress or Gateway API Intentionally

Ingress and Gateway API make Services reachable from outside the cluster. Both describe routing, but the concrete implementation depends on the installed controller.

Ingress example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-api
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shop-api
                port:
                  number: 80

Best practice

• Document which Ingress or Gateway controller is used.
• Define TLS, hostnames, and certificate management explicitly.
• Avoid controller-specific annotations without comments.
• Validate routing, TLS, backend Service, and Endpoints separately.

6.4. Isolate Traffic with NetworkPolicies

Without NetworkPolicies, Pod-to-Pod communication is broadly open in many clusters. NetworkPolicies define allowed traffic at the level of Pods, namespaces, IP blocks, and ports. Important: these rules only take effect if the CNI plugin supports NetworkPolicy enforcement.

Default deny for ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: shop-dev
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Allow traffic from the Ingress Controller to the API

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: shop-api-allow-ingress-controller
  namespace: shop-dev
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: shop-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

Best practice

• Start with default-deny per namespace.
• Allow only necessary communication paths.
• Document dependencies between services.
• Actively test policies with debug Pods.
• Also consider DNS egress when you restrict egress traffic.

7. Configuration, Secrets, and Storage

7.1. Separate Configuration from the Image

Container images should move through environments with as few changes as possible. Environment-specific configuration belongs in ConfigMaps, Secrets, or external configuration systems.

ConfigMap example

apiVersion: v1
kind: ConfigMap
metadata:
  name: shop-api-config
data:
  LOG_LEVEL: "info"
  FEATURE_CHECKOUT_V2: "true"

Use as environment variables

containers:
  - name: shop-api
    envFrom:
      - configMapRef:
          name: shop-api-config

Best practice

• Build an image once and use it across multiple environments.
• Do not bake configuration values into the image.
• Distinguish public configuration from sensitive data.
• Document whether a ConfigMap change requires a rollout.

7.2. Treat Secrets as Highly Sensitive

Kubernetes Secrets are intended for sensitive values, but they are not a complete replacement for secrets management. You must consciously plan access, storage, rotation, and usage.

Best practice

• Do not store real secrets in plaintext in Git.
• Use external secret systems or encrypted Git workflows when secrets must be versioned.
• Strictly limit RBAC access to Secrets.
• Enable encryption at rest for Secrets when you operate the cluster yourself or can configure the option in a managed service.
• Use short lifetimes and rotation where possible.
• Prefer mounting Secrets as files when applications support rotation.
• Avoid Secrets in logs, events, error messages, and command-line arguments.

Secret example for training purposes only

apiVersion: v1
kind: Secret
metadata:
  name: shop-api-db-credentials
type: Opaque
stringData:
  username: example-user
  password: example-password

Important

This example must not be treated as a production pattern for real passwords. In real projects, use a secrets-management approach that fits your organization.

7.3. Plan Persistence Explicitly

Pods are disposable. Data that must survive restarts, rescheduling, or rollouts does not belong in the writable container filesystem.

Best practice

• Use PersistentVolumeClaims for persistent data.
• Check access mode, StorageClass, reclaim policy, backup, and restore.
• Use StatefulSets for stateful workloads with stable volumes.
• Define which data may be lost.
• Test restore, not just backup.

PVC example

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shop-api-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard

Review questions

• Which data is persistent?
• Who creates the volume?
• What happens when the PVC is deleted?
• How is it backed up?
• How is restore tested?
• Which performance requirements apply?

8. Security Best Practices

8.1. RBAC: Least Privilege Instead of Cluster Admin

RBAC controls who may read, create, modify, or delete which API resources. Overly broad permissions are one of the most common security issues in Kubernetes clusters.

Best practice

• Grant permissions role-based and minimally.
• Use Role and RoleBinding inside a namespace whenever possible.
• Use ClusterRole only for cluster-wide resources or deliberately reusable roles.
• Avoid cluster-admin for applications.
• Check permissions with kubectl auth can-i.
• Regularly review bindings to system:masters, cluster-admin, and broad groups.

Example: namespace-scoped read permissions

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: shop-dev
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: shop-dev
subjects:
  - kind: ServiceAccount
    name: diagnostics
    namespace: shop-dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

Validation

kubectl auth can-i get pods --as=system:serviceaccount:shop-dev:diagnostics -n shop-dev
kubectl auth can-i delete secrets --as=system:serviceaccount:shop-dev:diagnostics -n shop-dev

8.2. Restrict ServiceAccounts

Every Pod runs with a ServiceAccount. If you do not specify one, Kubernetes uses the default ServiceAccount of the namespace.

Best practice

• Create a dedicated ServiceAccount per application or component.
• Bind only the required permissions.
• Set automountServiceAccountToken: false when the application does not need to talk to the Kubernetes API.
• Do not use shared ServiceAccounts for many independent applications.

Example

apiVersion: v1
kind: ServiceAccount
metadata:
  name: shop-api
  namespace: shop-dev
automountServiceAccountToken: false

Review question

Why does this Pod need access to the Kubernetes API? If you do not have a specific answer, the token should not be mounted automatically.

8.3. Apply Pod Security Standards

Pod Security Standards define security levels for Pods. The three profiles are privileged, baseline, and restricted.

Best practice

• Use restricted where possible.
• Use baseline when applications do not yet meet all restrictions.
• Use privileged only for clearly justified infrastructure workloads.
• Enforce standards through namespace labels or admission policies.
• Start by checking violations with warn and audit before enforcing hard blocks in production namespaces.

Namespace labels

apiVersion: v1
kind: Namespace
metadata:
  name: shop-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Migration pattern

1. Set warn=restricted and audit=restricted.
2. Fix reported violations.
3. Set enforce=baseline or enforce=restricted.
4. Document justified exceptions.

8.4. Harden Containers

Containers should run with as few privileges as possible.

Recommended baseline SecurityContext

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 10001
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault

Best practice

• Do not run as root if it can be avoided.
• Remove Linux capabilities you do not need.
• Avoid privileged: true.
• Avoid host namespace access such as hostNetwork, hostPID, and hostIPC, except for infrastructure components with a clear justification.
• Use a read-only root filesystem when the application can handle it.
• Write temporary files to explicit volumes such as emptyDir.

Example temporary write area

spec:
  containers:
    - name: shop-api
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir: {}

8.5. Secure Images and the Supply Chain

A Kubernetes cluster is only as trustworthy as the artifacts it runs.

Best practice

• Use images from trusted registries.
• Pin images at least with versioned tags; for higher security requirements, also use digests.
• Avoid latest in production manifests.
• Scan images for known vulnerabilities.
• Use small, maintained base images.
• Sign images if your organization operates a signing chain.
• Remove build tools, package managers, and unnecessary shells from runtime images where possible.
• Use private registries with controlled access.

Good pattern

image: registry.example.com/shop-api:1.4.2@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Poor pattern

image: shop-api:latest
imagePullPolicy: Always

imagePullPolicy: Always does not solve the core problem of an imprecise tag. A mutable tag remains mutable.

8.6. Use Admission Control and Policy as Code

Admission controllers validate or mutate API requests before resources are persisted. This allows security and operational rules to be enforced centrally.

Typical policy rules

• No containers with privileged: true.
• No production images with :latest.
• Every container must have resource requests.
• Pods must run as non-root.
• Namespaces must have Pod Security labels.
• Ingress resources must use TLS.
• Only approved registries are allowed.
• Certain annotations or labels are mandatory.

Policy-as-code tools

Best practice

• Test policies first in audit or warn mode.
• Document exceptions with an expiration date and justification.
• Keep policies versioned.
• Test policies in CI before enforcing them in the cluster.

9. Observability and Operations

9.1. Distinguish Logs, Events, and Metrics

These data sources answer different questions.

Best practice

• Applications log to stdout and stderr.
• Logs include timestamp, level, service, environment, and correlation ID.
• Kubernetes events are always checked during failure analysis.
• Metrics are not reconstructed from logs when they can be measured directly.
• Critical changes to the Kubernetes API are audited.

9.2. Build Observable Applications

Kubernetes can only control reliably what your application exposes as a signal.

Minimum requirements for services

Recommended endpoints

/startup   -> Has the application completed initial startup?
/ready     -> Can the application serve traffic?
/healthz   -> Is the process basically alive?
/metrics   -> Metrics for monitoring, if intended

9.3. Standardize Troubleshooting

Troubleshooting should not consist of random commands. Work from outside to inside and from broad signals to specific details.

Standard workflow

1. Clarify scope: Which application, which namespace, since when, which change?
2. Inspect objects: Deployment, ReplicaSet, Pod, Service, EndpointSlice, Ingress/Gateway.
3. Inspect status: Conditions, events, restarts, images, Nodes.
4. Inspect logs: current logs and --previous after restarts.
5. Inspect networking: DNS, Service selector, Endpoints, NetworkPolicy.
6. Inspect resources: requests, limits, OOM, NodePressure, quotas.
7. Inspect RBAC: kubectl auth can-i and ServiceAccount.
8. Assess the change: rollout history, Git diff, CI/CD runs.
9. Validate the fix: document before-and-after commands.
10. Document rollback: how the fix can be reverted.

Core commands

kubectl get all -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl rollout status deployment/<deployment> -n <namespace>
kubectl rollout history deployment/<deployment> -n <namespace>

10. Deployment, Release, and Rollback

10.1. Use Reproducible Images

A Deployment should remain traceable later. You must know which code, dependencies, and image actually ran.

Best practice

• Build images in CI.
• Version images with immutable versions.
• Store Git commit, build time, and build pipeline as metadata.
• Use digests for especially critical workloads.
• Document who is allowed to promote an image.

Useful annotations

metadata:
  annotations:
    build.example.com/git-sha: "f4c2a7e"
    build.example.com/pipeline: "https://ci.example.com/pipelines/12345"

10.2. Test Rollouts and Rollbacks

A rollback path is only useful if it has been tested.

Check rollout

kubectl apply -f k8s/
kubectl rollout status deployment/shop-api
kubectl get pods -l app.kubernetes.io/name=shop-api

Check rollback

kubectl rollout history deployment/shop-api
kubectl rollout undo deployment/shop-api
kubectl rollout status deployment/shop-api

Best practice

• Test rollout and rollback before production changes.
• Review database migrations separately. Not every data change is rollback-capable.
• Document whether a rollback affects only application code or also data.
• Define clear abort criteria: error rate, latency, CrashLoop, business metrics.

10.3. Use CI/CD Quality Gates

Manifests should not fail for the first time in the cluster.

Recommended pipeline steps

1. YAML linting
2. Kubernetes schema validation
3. Policy checks
4. Image scan
5. Secret scan
6. Unit and integration tests
7. Render Helm or Kustomize
8. Diff against the target environment
9. Deployment to a test environment
10. Smoke test
11. Approval or automated promotion process

Example gates

11. Review Checklists

11.1. Manifest Review

11.2. Security Review

11.3. Operations Review

11.4. Final Project Review

For a final project, a solution is considered complete when you can demonstrate the following points:

• You can deploy the application from the repository.
• You can explain which resources are created.
• You can reach the application internally or externally.
• You can inspect logs, events, and status.
• You can trigger and observe a rollout.
• You can perform a rollback.
• You have set requests, limits, and probes.
• You have used a dedicated namespace.
• You have no real secrets in the repository.
• You have implemented at least one security measure: RBAC, Pod Security, SecurityContext, or NetworkPolicy.
• You have intentionally created and resolved at least one common failure mode.

12. Anti-Patterns and Failure Modes

12.1. Pod Remains Pending

Typical causes

• Not enough CPU or memory in the cluster.
• Node selector, affinity, or taints prevent scheduling.
• PVC is not bound.
• ResourceQuota is exhausted.
• Image pulling is not relevant yet because the Pod has not been scheduled.

Validation

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe quota -n <namespace>
kubectl get nodes
kubectl describe node <node>

Correction

• Adjust requests realistically.
• Check or extend quotas.
• Correct affinity and tolerations.
• Check PVC and StorageClass.
• Increase cluster capacity.

12.2. CrashLoopBackOff

Typical causes

• The application exits immediately.
• Configuration is missing or incorrect.
• Secret or ConfigMap is missing.
• LivenessProbe is too aggressive.
• The application cannot reach a dependency.
• Startup command or arguments are wrong.

Validation

kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Correction

• Read logs, do not merely count restarts.
• Check ConfigMaps, Secrets, and environment variables.
• Relax probes or add a StartupProbe.
• Reproduce the container locally or in a debug environment.

12.3. ImagePullBackOff or ErrImagePull

Typical causes

• Image name or tag is wrong.
• Registry is not reachable.
• Authentication is missing.
• imagePullSecret is wrong or in the wrong namespace.
• Rate limits or network problems prevent the pull.

Validation

kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
kubectl get serviceaccount <sa> -n <namespace> -o yaml

Correction

• Verify the exact image reference.
• Test registry access.
• Create imagePullSecrets in the correct namespace.
• Reference the ServiceAccount correctly.
• Avoid latest to improve diagnosis and reproducibility.

12.4. Service Without Endpoints

Typical causes

• Service selector does not match Pod labels.
• Pods are not Ready.
• Wrong namespace.
• Wrong targetPort.

Validation

kubectl get svc <service> -n <namespace> -o yaml
kubectl get pods -n <namespace> --show-labels
kubectl get endpoints <service> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>

Correction

• Align labels and selector.
• Check the ReadinessProbe.
• Check port names and targetPort.
• Correct the namespace.

12.5. DNS Does Not Work Inside the Cluster

Typical causes

• Wrong Service name or namespace.
• CoreDNS is not running correctly.
• NetworkPolicy blocks DNS.
• The Service has no Endpoints, even though DNS resolves.

Validation

kubectl run dns-debug --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
kubectl run dns-debug --rm -it --image=busybox:1.36 --restart=Never -- nslookup <service>.<namespace>.svc.cluster.local
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Correction

• Correct DNS names.
• Check CoreDNS.
• Allow egress to DNS when egress policies are active.
• Check Service Endpoints separately.

12.6. Ingress or Gateway Does Not Route

Typical causes

• IngressClass or GatewayClass is missing.
• Controller is not running.
• Host or path is wrong.
• Backend Service has no Endpoints.
• TLS Secret is missing or incorrect.
• Controller-specific annotation is wrong.

Validation

kubectl describe ingress <name> -n <namespace>
kubectl get ingressclass
kubectl get svc <backend-service> -n <namespace>
kubectl get endpoints <backend-service> -n <namespace>
kubectl logs -n <controller-namespace> deploy/<controller-deployment>

Correction

• Check routing rules.
• Read controller logs.
• Repair backend Service and Endpoints.
• Validate TLS Secret and hostnames.

12.7. RBAC Forbidden

Typical causes

• User or ServiceAccount has no matching RoleBinding.
• Role is bound in the wrong namespace.
• Resource belongs to a different API group.
• Verb is missing: get, list, watch, create, update, patch, delete.

Validation

kubectl auth can-i get pods -n <namespace>
kubectl auth can-i get secrets -n <namespace> --as=system:serviceaccount:<namespace>:<serviceaccount>
kubectl get rolebinding,clusterrolebinding -A | grep <name>

Correction

• Determine the required verbs and resources exactly.
• Create the RoleBinding in the correct namespace.
• Do not use a broad ClusterRoleBinding if a Role is sufficient.

12.8. NodePressure

Typical causes

• Too little free disk space.
• Too little memory.
• Too many logs or temporary files.
• Images consume too much space.
• Too many Pods on one Node.

Validation

kubectl describe node <node>
kubectl get events -A --sort-by=.lastTimestamp
kubectl top nodes
kubectl top pods -A

Correction

• Check requests and limits.
• Limit ephemeral storage.
• Check log rotation.
• Remove unused images and Pods.
• Increase Node capacity or distribute workloads.

12.9. OOMKilled

Typical causes

• Memory limit is too low.
• Application has a memory leak.
• Peak load was not considered.
• Cache grows without bounds.

Validation

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState}'
kubectl top pod <pod> -n <namespace>

Correction

• Measure the memory profile.
• Adjust the limit realistically.
• Analyze memory leaks.
• Bound cache growth.
• Set alerts for memory usage and restarts.

12.10. PVC Remains Pending

Typical causes

• StorageClass does not exist.
• Dynamic provisioning does not work.
• Access mode is not supported.
• Zone or Node affinity does not match.
• Quota prevents provisioning.

Validation

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Correction

• Check StorageClass.
• Check CSI driver.
• Adjust access mode.
• Check quotas.
• Inspect provider-specific errors in the controller.

12.11. HPA Does Not Scale

Typical causes

• Metrics Server is missing or does not deliver data.
• CPU requests are missing.
• Target metric is defined incorrectly.
• minReplicas and maxReplicas prevent the expected behavior.
• Load does not reach the target value.

Validation

kubectl get hpa -n <namespace>
kubectl describe hpa <hpa> -n <namespace>
kubectl top pods -n <namespace>
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

Correction

• Deploy or repair Metrics Server.
• Set requests.
• Adjust metric and target value.
• Run a load test.
• Check scaling boundaries.

13. Workshop Exercises

Exercise 1: Review a Deployment

Task

You receive a Deployment without requests, probes, or SecurityContext. Rework it so it is suitable for a workshop and can be reviewed properly.

Acceptance criteria

• Every container has CPU and memory requests.
• A memory limit is set.
• A ReadinessProbe is present.
• A LivenessProbe is justified.
• The container runs as non-root.
• Privilege escalation is disabled.
• The image is versioned.

Exercise 2: Debug a Service Without Endpoints

Initial situation

A Service exists, but the application is not reachable.

Task

Find out whether the problem is caused by DNS, the Service selector, readiness, or Pod labels.

Required commands

kubectl get svc -o yaml
kubectl get pods --show-labels
kubectl get endpoints
kubectl describe pod <pod>

Result

Document the cause, fix, and validation.

Exercise 3: Grant Minimal RBAC Permissions

Task

Create a ServiceAccount that may only read Pods and Pod logs in a namespace.

Acceptance criteria

• No ClusterRoleBinding.
• No write permissions.
• Validation with kubectl auth can-i.

Exercise 4: Introduce NetworkPolicy

Task

First set default-deny in a namespace, then allow only traffic from the Ingress Controller to the API.

Acceptance criteria

• Unauthorized Pod-to-Pod traffic is blocked.
• Allowed HTTP traffic works.
• DNS still works if egress has been restricted.

Exercise 5: Test Rollout and Rollback

Task

Roll out a new image version, observe the rollout, and then perform a rollback.

Required commands

kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>

Result

Document how you identified a successful rollout and rollback.

14. Operational Readiness

A solution is operationally ready when it is reproducible, observable, reversible, reviewable, and free of real secrets. For Kubernetes, however, this short definition is not enough. Use the following maturity matrix.

Minimum definition for workshop submissions

A workshop submission is considered operationally ready when you can demonstrate the following points:

• Deployment, Service, and Namespace are defined declaratively.
• Resources are grouped traceably with labels.
• Images are versioned.
• Requests and sensible limits are set.
• A ReadinessProbe is present.
• Logs and events can be inspected.
• Rollout and rollback are documented.
• No real secrets are included.
• A dedicated ServiceAccount is used or consciously not required.
• At least one NetworkPolicy or SecurityContext hardening measure is implemented.
• Common failure modes are documented.

15. Command Reference

Context and Namespace

kubectl config current-context
kubectl config get-contexts
kubectl get namespaces
kubectl config set-context --current --namespace=<namespace>

Show Resources

kubectl get all -n <namespace>
kubectl get pods -n <namespace> -o wide
kubectl get deployment,rs,pod,svc,endpointslice -n <namespace>

Details and Events

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Logs

kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl logs deployment/<deployment> -n <namespace>

Rollout

kubectl rollout status deployment/<deployment> -n <namespace>
kubectl rollout history deployment/<deployment> -n <namespace>
kubectl rollout undo deployment/<deployment> -n <namespace>

Manifest Validation

kubectl apply --dry-run=server -f k8s/
kubectl diff -f k8s/
kubectl apply -f k8s/

RBAC

kubectl auth can-i get pods -n <namespace>
kubectl auth can-i get secrets -n <namespace> --as=system:serviceaccount:<namespace>:<serviceaccount>

Network Debugging

kubectl run net-debug --rm -it --image=busybox:1.36 --restart=Never -- sh
nslookup <service>.<namespace>.svc.cluster.local
wget -S -O- http://<service>.<namespace>.svc.cluster.local

Resources

kubectl top nodes
kubectl top pods -A
kubectl describe quota -n <namespace>
kubectl describe limitrange -n <namespace>

16. Sources and Further Reading

This material is based on the official Kubernetes documentation and extends it with practical review and workshop guidance.

• Kubernetes Documentation: https://kubernetes.io/docs/home/
• Kubernetes API and declarative object management: https://kubernetes.io/docs/concepts/overview/kubernetes-api/
• kubectl apply: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_apply/
• kubectl diff: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_diff/
• Resource management for Pods and containers: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
• Liveness, Readiness, and Startup Probes: https://kubernetes.io/docs/concepts/workloads/pods/probes/
• Services: https://kubernetes.io/docs/concepts/services-networking/service/
• Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
• Gateway API: https://kubernetes.io/docs/concepts/services-networking/gateway/
• NetworkPolicies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• RBAC Good Practices: https://kubernetes.io/docs/concepts/security/rbac-good-practices/
• ServiceAccounts: https://kubernetes.io/docs/concepts/security/service-accounts/
• Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
• Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/
• Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
• Good Practices for Kubernetes Secrets: https://kubernetes.io/docs/concepts/security/secrets-good-practices/
• Images: https://kubernetes.io/docs/concepts/containers/images/
• Horizontal Pod Autoscaling: https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/
• PodDisruptionBudgets: https://kubernetes.io/docs/tasks/run-application/configure-pdb/
• Logging Architecture: https://kubernetes.io/docs/concepts/cluster-administration/logging/
• Auditing: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/