1. Objective and Scope

1.1. Objective

After working through this material, you should be able to do more than start a Kubernetes cluster: you should be able to operate it. This includes the ability to:

• detect cluster, node, network, storage, and workload problems early;
• perform changes in a controlled and reversible way;
• back up cluster state, application data, configuration, certificates, and operational artifacts;
• test restore and disaster-recovery procedures regularly;
• patch operating systems, container runtimes, Kubernetes components, and add-ons;
• plan Kubernetes minor and patch upgrades without unnecessary outage risk;
• operate monitoring, logging, tracing, and alerting as a coherent observability system;
• distinguish between local lab environments, self-managed clusters, and managed Kubernetes.

1.2. Scope Boundaries

This document does not replace provider-specific product documentation or internal company runbooks. It provides a robust reference model and concrete runbooks that you can adapt to these environments:

• WSL2 on Windows 11;
• kind, minikube, and k3d;
• VirtualBox, VMware Workstation, VMware Fusion, and Vagrant VMs;
• rented remote VMs;
• bare metal;
• kubeadm clusters;
• k3s, RKE2, k0s, and Talos;
• managed Kubernetes such as EKS, AKS, GKE, DOKS, LKE, OKE, IKS, and OpenShift-based platforms.

1.3. Assumptions

• The authoritative desired state is stored in Git, an IaC repository, a GitOps repository, or an equivalent versioned source.
• Changes are not tested directly in a production cluster.
• A backup is only usable once at least one restore test has succeeded.
• Monitoring without alerting is only visibility. Alerting without a runbook creates noise. Logging without a search and retention concept is not an operations concept.
• Day-2 Operations are not only technical. They consist of processes, responsibilities, maintenance windows, communication paths, and evidence.

2. Day-2 Operating Model

2.1. What Day-2 Operations Means

Day 0 is planning. Day 1 is provisioning. Day 2 is operations. In Kubernetes, Day-2 Operations includes all recurring and event-driven tasks after initial deployment.

2.2. Operational Principles

1. Observe first, then change. Capture the starting state before you act.
2. Every change needs a way back. The return path can be rollback, restore, rebuild, or deactivation.
3. Cluster state is not application state. Kubernetes objects and persistent data must be evaluated separately.
4. Automation does not replace understanding. Automated upgrades, GitOps, and autoscalers need limits, alerts, and runbooks.
5. Providers take tasks off your plate, but not accountability. Managed Kubernetes reduces control-plane operations, but workloads, add-ons, cost, security, backups, and dependencies remain your responsibility.

2.3. Service Classes

Not every environment needs the same operational intensity. Define service classes before promising runbooks and SLAs.

2.4. RTO, RPO, and Operational Targets

Important: A Kubernetes backup without an RTO/RPO statement is only a technical activity. It becomes an operations concept only when recovery objectives, test intervals, and ownership are defined.

3. Operations Architecture

3.1. Overall View

flowchart TD
    Users[Users and Systems] --> Edge[DNS, Load Balancer, Ingress, Gateway]
    Edge --> Services[Kubernetes Services]
    Services --> Workloads[Pods and Workloads]
    Workloads --> Storage[Persistent Storage]
    Workloads --> Config[ConfigMaps, Secrets, External Configuration]

    Operators[Platform / SRE / DevOps] --> Git[Git, IaC, GitOps]
    Git --> CD[CI/CD or GitOps Controller]
    CD --> API[Kubernetes API Server]
    API --> State[etcd / Managed Control Plane State]
    API --> Controllers[Controllers, Scheduler, Operators]
    Controllers --> Workloads

    Workloads --> Logs[Logs]
    Workloads --> Metrics[Metrics]
    Workloads --> Traces[Traces]
    API --> Audit[Audit Logs]
    Logs --> Obs[Observability Backend]
    Metrics --> Obs
    Traces --> Obs
    Audit --> Obs
    Obs --> Alerts[Alertmanager / Cloud Alerts / On-Call]
    Alerts --> Operators

    State --> Backup[Backups]
    Storage --> Backup
    Git --> Backup
    Backup --> Restore[Restore and DR]
    Restore --> API

3.2. Operations Loop

flowchart LR
    Plan[Plan] --> Change[Prepare Change]
    Change --> Validate[Dry Run, Tests, Policy Checks]
    Validate --> Execute[Execute]
    Execute --> Observe[Observe]
    Observe --> Decide{Target reached?}
    Decide -- Yes --> Document[Document]
    Decide -- No --> Mitigate[Rollback, Rollforward, or Restore]
    Mitigate --> Observe
    Document --> Improve[Improve Runbook]
    Improve --> Plan

3.3. Three State Layers

flowchart TD
    Desired[Desired State: Git, Helm, Kustomize, Terraform, Crossplane] --> API[Kubernetes API]
    API --> Cluster[Cluster State: etcd, API Objects, Controller Status]
    Cluster --> Runtime[Runtime State: Pods, Containers, Volumes, Network]
    Runtime --> Data[Data State: Databases, PVCs, Object Storage, External Systems]
    Data --> Backup[Backup and Replication]
    Backup --> Restore[Restore Test]
    Restore --> Desired

Operations are stable when all three layers are understood:

• Desired State: What should run?
• Cluster State: What does Kubernetes think should run?
• Runtime/Data State: What actually runs, and which data is critical?

4. Environment Matrix

4.1. Environment Types

4.2. Decision Matrix for Operational Effort

4.3. Not Every Lab Needs Production Complexity

For workshops, it is useful to demonstrate concepts without artificially forcing production-grade processes. You can use the same structure, but adjust depth:

• kind/k3d: Rebuild is often faster and cleaner than restore.
• minikube: Deleting and recreating the profile is often the best reset.
• WSL: Operations focus on tooling, port access, local hosts files, and clean filesystem separation.
• VM/Bare Metal/kubeadm: real backups, etcd snapshots, node maintenance, and upgrade ordering matter.
• Managed Kubernetes: the provider handles parts of the control plane; you remain responsible for workloads, add-ons, security, observability, data, and cost.

5. Responsibility Model

5.1. Self-Managed vs. Managed Kubernetes

flowchart TD
    subgraph Self[Self-Managed Cluster]
      A1[Control Plane] --> A2[etcd]
      A2 --> A3[Nodes]
      A3 --> A4[CNI, CSI, Ingress]
      A4 --> A5[Workloads]
      A5 --> A6[Backups and Monitoring]
    end

    subgraph Managed[Managed Kubernetes]
      P1[Provider Control Plane] --> P2[Provider Upgrades]
      P2 --> P3[Node Pools]
      P3 --> P4[Add-ons]
      P4 --> P5[Workloads]
      P5 --> P6[Backups and Monitoring]
    end

5.2. RACI Template

Legend: A = Accountable, R = Responsible, C = Consulted, I = Informed.

6. Operational Foundations Before the First Incident

6.1. Minimum Standard per Cluster

A cluster is not production-ready just because kubectl get nodes looks green. For Day-2 Operations, you need at least:

• documented cluster ID, environment, owner, criticality, and cost center;
• supported Kubernetes version and target upgrade path;
• working kubeconfig with role-based access;
• Git repository for manifests, Helm values, Kustomize overlays, or GitOps;
• defined namespaces and owners;
• resource requests, limits, quotas, and LimitRanges;
• liveness, readiness, and startup probes for relevant workloads;
• PodDisruptionBudgets for highly available workloads;
• NetworkPolicies for tenant separation and default-deny models;
• central metrics, logs, and events;
• alerts with severity, runbook link, and owner;
• backup plan with retention and restore testing;
• upgrade and maintenance windows;
• decommissioning and cost controls.

6.2. Cluster Inventory

Maintain an inventory. It can be kept as Markdown, CMDB, Terraform output, Backstage catalog, or GitOps metadata.

6.3. Kubeconfig and Access Concept

• Use separate contexts for dev, stage, and prod.
• Do not grant permanent cluster-admin access for daily work.
• Use kubectl auth can-i before widening permissions.
• Check whether kubeconfig files end up in repositories, tickets, chat history, or artifacts.
• Use short-lived tokens, OIDC, cloud IAM, or central identities where possible.

kubectl config get-contexts
kubectl config current-context
kubectl auth can-i get pods -A
kubectl auth can-i '*' '*' --as system:serviceaccount:demo:app
kubectl get clusterrolebinding -o wide

6.4. Namespace Baseline

Every production-like namespace should have a baseline:

• owner label;
• environment label;
• ResourceQuota;
• LimitRange;
• Pod Security Admission label;
• NetworkPolicy default deny;
• optional egress policy;
• ServiceAccount per application;
• runbook link as annotation.

apiVersion: v1
kind: Namespace
metadata:
  name: demo
  labels:
    environment: stage
    owner: team-platform
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
  annotations:
    runbook.example.com/url: https://runbooks.example.local/kubernetes/demo
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: demo-quota
  namespace: demo
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "40"
    persistentvolumeclaims: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: demo-defaults
  namespace: demo
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        cpu: 500m
        memory: 512Mi
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: demo
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

7. Standard Runbook for Every Change

7.1. Process

1. Define scope: cluster, namespace, workload, add-on, node pool, or provider resource.
2. Assess risk: impact, downtime, data loss, security risk, cost risk.
3. Capture starting state: versions, Pods, Events, metrics, logs, backups, open alerts.
4. Check backup: Is a backup or snapshot required before the change?
5. Prepare change: manifest, Helm values, Terraform plan, CLI command, GitOps PR.
6. Pre-validate: kubectl diff, --dry-run=server, Helm template, policy checks, staging test.
7. Communicate: maintenance window, affected parties, on-call, rollback decision maker.
8. Execute: perform the change in small steps.
9. Observe: rollout, Events, logs, SLOs, error rates, latency, resources, costs.
10. Decide: stable, roll forward, roll back, restore, or incident.
11. Document: result, deviations, new risks, runbook improvements.

7.2. Before/After Commands

# Context and version
kubectl config current-context
kubectl version
kubectl get nodes -o wide

# Cluster state
kubectl get componentstatuses 2>/dev/null || true
kubectl get --raw='/readyz?verbose' 2>/dev/null || true
kubectl get --raw='/livez?verbose' 2>/dev/null || true
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -100

# Target namespace
NS=demo
kubectl get all -n "$NS" -o wide
kubectl get deploy,sts,ds,job,cronjob -n "$NS"
kubectl get configmap,secret,sa,role,rolebinding -n "$NS"
kubectl get pvc -n "$NS"

# Manifest validation
kubectl apply --dry-run=server -f manifest.yaml
kubectl diff -f manifest.yaml

# After
kubectl rollout status deployment/demo -n "$NS" --timeout=180s
kubectl get pods -n "$NS" -o wide
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -50

7.3. Change Types

8. Daily, Weekly, and Monthly Routines

8.1. Daily Routine

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -80
kubectl top nodes 2>/dev/null || echo "metrics-server unavailable"
kubectl get volumesnapshot -A 2>/dev/null || true
velero backup get 2>/dev/null || true

8.2. Weekly Routine

• Review top alerts and most frequent Events.
• Check outdated images and add-on versions.
• Check Pods without requests/limits.
• Check namespaces without quotas and NetworkPolicies.
• Check PVCs with high utilization.
• Test a representative restore in a non-production environment.
• Review cost, unused LoadBalancers, PVs, and node pools.
• Check certificate expiration dates.

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{":cpu="}{.resources.requests.cpu}{",mem="}{.resources.requests.memory}{" "}{end}{"\n"}{end}'
kubectl get networkpolicy -A
kubectl get resourcequota -A
kubectl get limitrange -A
kubectl get certificates -A 2>/dev/null || true
kubectl get ingress -A
kubectl get svc -A --field-selector spec.type=LoadBalancer 2>/dev/null || kubectl get svc -A

8.3. Monthly Routine

• Review Kubernetes release and provider notices.
• Update the minor/patch upgrade plan.
• Run the DR runbook against a test environment.
• Perform RBAC and ServiceAccount review.
• Check backup retention and encryption.
• Review dashboard and alert relevance.
• Evaluate capacity forecast and cost trend.
• Check CRDs, webhooks, and operators for version and API compatibility.
• Patch node images, kernel, and container runtime or rotate node pools.

8.4. Quarterly Routine

• Full restore test of a production-like workload.
• Incident drill with realistic symptoms.
• Review RTO/RPO targets.
• Review emergency access and break-glass procedures.
• Review audit logs and security findings.
• Upgrade rehearsal for the next Kubernetes minor release.
• Check whether deprecated API versions must be removed.

9. Cluster Health and Triage

9.1. Fast Triage in 5 Minutes

# 1. Confirm context
kubectl config current-context

# 2. Nodes and versions
kubectl get nodes -o wide
kubectl version

# 3. Problematic Pods
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get pods -A | egrep 'CrashLoopBackOff|ImagePullBackOff|ErrImagePull|Pending|Evicted|OOMKilled|CreateContainerConfigError' || true

# 4. Events
kubectl get events -A --sort-by=.lastTimestamp | tail -120

# 5. Control plane readiness
kubectl get --raw='/readyz?verbose' 2>/dev/null || true
kubectl get --raw='/livez?verbose' 2>/dev/null || true

# 6. Resources
kubectl top nodes 2>/dev/null || true
kubectl top pods -A 2>/dev/null || true

# 7. Add-ons
kubectl get pods -n kube-system -o wide

9.2. Control Plane Health

With managed Kubernetes, the control plane is only partially visible. With kubeadm, bare metal, VM, and remote VM clusters, also check system services and static Pods.

# kubeadm / self-managed control plane
sudo crictl ps | egrep 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'
sudo crictl logs $(sudo crictl ps --name kube-apiserver -q | head -1)
sudo journalctl -u kubelet -n 200 --no-pager
sudo ls -l /etc/kubernetes/manifests
sudo ls -l /var/lib/etcd

# Check API internally
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'
kubectl get --raw='/metrics' | head

9.3. Node Health

NODE=<node-name>
kubectl describe node "$NODE"
kubectl get pods -A -o wide --field-selector spec.nodeName="$NODE"
kubectl top node "$NODE" 2>/dev/null || true

# On the node
sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
sudo crictl ps -a
sudo crictl images
sudo crictl stats
sudo df -h
sudo free -h
sudo ip route
sudo ss -tulpn

9.4. Reading Events Correctly

Events are short-lived operational indicators. They do not replace logs, but often reveal the cause.

10. Workload Operations

10.1. Minimum Standard for Production-Ready Workloads

A workload is operationally ready when it satisfies at least these points:

• Deployment, StatefulSet, DaemonSet, Job, or CronJob is deliberately chosen.
• Container images are versioned and not just latest.
• Requests and limits are set.
• Readiness probe protects traffic.
• Liveness probe is not more aggressive than the application can tolerate.
• Startup probe protects slow-starting applications.
• Graceful shutdown is covered via terminationGracePeriodSeconds and signal handling.
• PodDisruptionBudget protects against voluntary disruptions.
• ConfigMaps and Secrets are separated.
• Logs go to stdout/stderr and do not contain Secrets.
• ServiceAccount has minimum permissions.
• SecurityContext is set.
• Rollout and rollback are tested.

10.2. Example: Operationally Ready Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-api
  namespace: demo
  labels:
    app.kubernetes.io/name: demo-api
    app.kubernetes.io/part-of: workshop
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: demo-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: demo-api
    spec:
      serviceAccountName: demo-api
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: ghcr.io/example/demo-api:1.2.3
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          envFrom:
            - configMapRef:
                name: demo-api-config
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: demo-api
  namespace: demo
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: demo-api

10.3. Monitor a Rollout

NS=demo
APP=demo-api
kubectl rollout status deployment/$APP -n $NS --timeout=180s
kubectl rollout history deployment/$APP -n $NS
kubectl get rs -n $NS -l app.kubernetes.io/name=$APP
kubectl get pods -n $NS -l app.kubernetes.io/name=$APP -o wide
kubectl logs deployment/$APP -n $NS --all-containers --tail=100

10.4. Perform a Rollback

NS=demo
APP=demo-api
kubectl rollout undo deployment/$APP -n $NS
kubectl rollout status deployment/$APP -n $NS --timeout=180s
kubectl rollout history deployment/$APP -n $NS

With GitOps, a manual rollback in the cluster is only an immediate mitigation. Afterwards, the authoritative state in Git must be corrected; otherwise GitOps will reconcile the faulty change again.

10.5. Check Workload Drift

# Export server state without blindly trusting runtime-generated fields
kubectl get deployment demo-api -n demo -o yaml

# Compare against Git
kubectl diff -f k8s/overlays/stage/

# Argo CD
argocd app get demo-api
argocd app diff demo-api

# Flux
flux get kustomizations -A
flux diff kustomization demo-api --path ./clusters/stage/demo-api

11. Monitoring Concept

11.1. Target State

Monitoring should not produce as many graphs as possible. It should answer:

• Is the cluster reachable?
• Are critical workloads available?
• Are nodes, Pods, or PVCs consuming critical resources?
• Is an SLO breach imminent?
• Are backups current?
• Are rollouts or GitOps syncs stuck?
• Do DNS, Ingress, CNI, and CSI work?
• Are there security or compliance violations?

flowchart TD
    KubeAPI[Kubernetes API] --> Prom[Prometheus]
    Kubelet[kubelet and cAdvisor] --> Prom
    NodeExporter[Node Exporter] --> Prom
    KSM[kube-state-metrics] --> Prom
    App[Application Metrics] --> Prom
    Blackbox[Blackbox Exporter] --> Prom
    Prom --> Rules[Recording and Alerting Rules]
    Rules --> Alertmanager[Alertmanager]
    Prom --> Grafana[Grafana Dashboards]
    Alertmanager --> OnCall[On-Call, Chat, Ticket]
    Logs[Loki/OpenSearch/Cloud Logs] --> Grafana
    Traces[Tempo/Jaeger/Cloud Trace] --> Grafana

11.2. Metric Sources

11.3. Minimal Prometheus Stack

For production-like clusters, kube-prometheus-stack is a common starting point because it bundles Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter, and useful default rules. In workshops, install it via Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.enabled=true \
  --set prometheus.prometheusSpec.retention=7d

kubectl get pods -n monitoring
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

11.4. Monitoring by Environment

11.5. Minimum Dashboard Set

• Cluster Overview: API, nodes, Pods, errors, capacity.
• Namespace Overview: CPU, memory, Pods, restarts, quotas.
• Workload Overview: requests, limits, HPA, rollouts, latency, error rate.
• Node Overview: CPU, memory, disk, network, kubelet, runtime.
• Storage Overview: PVC utilization, volume attach/mount, snapshot status.
• Network Overview: DNS latency, Ingress/Gateway, CNI flows, drops.
• Backup Overview: last success, duration, retention, restore-test status.
• Security Overview: policy violations, admission denials, audit anomalies.

11.6. Example: ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-api
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - demo
  selector:
    matchLabels:
      app.kubernetes.io/name: demo-api
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

12. Logging Concept

12.1. Target State

Kubernetes stores container logs on nodes and exposes them through kubectl logs. This is not sufficient for production because Pods, containers, and nodes are ephemeral. Cluster-level logging needs an agent per node and a separate backend with search, retention, and access concepts.

flowchart TD
    PodA[Pod stdout/stderr] --> NodeLog[/Node Log Files/]
    PodB[Pod stdout/stderr] --> NodeLog
    NodeLog --> Agent[Fluent Bit, Vector, Promtail, Grafana Alloy]
    Agent --> Backend[Loki, Elasticsearch, OpenSearch, Cloud Logging, CloudWatch Logs, Azure Monitor]
    Backend --> Query[Search, Dashboards, Correlation]
    Query --> Incident[Incident Analysis]

12.2. Logging Rules for Applications

• Write logs to stdout/stderr.
• Use structured logs, preferably JSON.
• Does every log entry include time, level, service, version, trace ID, request ID, and relevant labels?
• Do not write passwords, tokens, personal data, or private keys to logs.
• Logically separate audit logs, access logs, and application logs.
• Define retention according to criticality and compliance.
• Use sampling for very noisy debug logs.

12.3. Logging Backends

12.4. Example: Loki with Helm

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki \
  --namespace logging --create-namespace
helm upgrade --install promtail grafana/promtail \
  --namespace logging \
  --set "config.clients[0].url=http://loki-gateway.logging.svc.cluster.local/loki/api/v1/push"

kubectl get pods -n logging

12.5. Log Retention

12.6. Log Triage

# Kubernetes-native
kubectl logs deployment/demo-api -n demo --all-containers --tail=200
kubectl logs pod/demo-api-abc -n demo -c app --previous
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

# Common searches in Loki LogQL
# {namespace="demo", app="demo-api"} |= "error"
# {namespace="ingress-nginx"} | json | status >= 500
# {namespace="demo"} |= "trace_id="

13. Tracing and OpenTelemetry

13.1. Why Tracing Belongs to Day-2

Metrics show that something is slow or failing. Logs explain individual events. Traces show paths across services. In distributed Kubernetes applications, tracing is especially useful for:

• latency between microservices;
• errors at external dependencies;
• database and queue wait times;
• correlation between Ingress, Service, and backend;
• root-cause analysis for sporadic errors.

13.2. Target Architecture

flowchart LR
    App1[Service A] --> OTel[OpenTelemetry Collector]
    App2[Service B] --> OTel
    App3[Service C] --> OTel
    OTel --> Tempo[Tempo or Jaeger]
    OTel --> Prom[Prometheus or Mimir]
    OTel --> Logs[Loki or Log Backend]
    Tempo --> Grafana[Grafana]
    Prom --> Grafana
    Logs --> Grafana

13.3. Operational Rules

• Use consistent resource attributes: service.name, service.version, deployment.environment, k8s.namespace.name, k8s.pod.name.
• Enforce trace-context propagation across HTTP, gRPC, and messaging.
• Keep sampling deliberately controlled.
• Do not export telemetry data across country or tenant boundaries without control.
• Correlation means: metrics, logs, and traces must carry the same service and request context.

13.4. Minimal Collector as Concept Example

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    processors:
      batch: {}
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
    exporters:
      logging:
        verbosity: basic
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging]

14. Alerting Concept

14.1. Alerting Goal

An alert should trigger a human or automated action. Alerts that require no action belong in dashboards, reports, or tickets, but not in night-time on-call.

14.2. Alert Severities

14.3. Good Alerts

A good alert contains:

• What: precise condition;
• Where: cluster, namespace, workload, node, region;
• Since when: duration and trend;
• Impact: SLO, customer impact, data risk;
• Owner: team or service;
• Runbook: link or clear next steps;
• Labels: severity, team, service, environment, cluster.

14.4. Example: PrometheusRule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-day2-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: kubernetes.day2.workloads
      rules:
        - alert: KubernetesPodCrashLooping
          expr: max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[10m]) >= 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is in CrashLoopBackOff"
            runbook_url: "https://runbooks.example.local/kubernetes/pod-crashloopbackoff"
        - alert: KubernetesDeploymentReplicasMismatch
          expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} does not have enough available replicas"
        - alert: KubernetesPersistentVolumeClaimFillingUp
          expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.85
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is more than 85 percent full"
        - alert: KubernetesNodeNotReady
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} is not Ready"
        - alert: KubernetesBackupMissing
          expr: time() - max(kubernetes_backup_last_success_timestamp_seconds) > 86400
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Latest successful Kubernetes backup is older than 24 hours"

14.5. Alertmanager Routing

route:
  receiver: default
  group_by: ['cluster', 'namespace', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: oncall-critical
      repeat_interval: 30m
    - matchers:
        - severity="warning"
      receiver: team-warning
      repeat_interval: 4h
receivers:
  - name: default
    webhook_configs:
      - url: http://alert-router.monitoring.svc.cluster.local/default
  - name: oncall-critical
    webhook_configs:
      - url: http://alert-router.monitoring.svc.cluster.local/critical
  - name: team-warning
    webhook_configs:
      - url: http://alert-router.monitoring.svc.cluster.local/warning

14.6. Alerts to Avoid

• CPU > 80% for individual Pods without context. CPU may be high while SLOs remain stable.
• Paging on every single Pod restart. Short-lived restarts are not always critical.
• Alerts without a for duration.
• Alerts on raw metrics without owner and runbook.
• Alerts on local labs with production severity.
• Duplicate alerts from Prometheus and cloud monitoring without deduplication.

15. Backup Concept

15.1. What You Must Back Up

flowchart TD
    Git[Git: Manifests, Helm, Kustomize, GitOps] --> Recovery[Recovery]
    IaC[IaC: Terraform/OpenTofu/Pulumi State] --> Recovery
    Etcd[etcd or Managed Control Plane State] --> Recovery
    PV[Persistent Volumes and Snapshots] --> Recovery
    DB[Application Databases] --> Recovery
    Secrets[Secrets, Certificates, Key Material] --> Recovery
    Registry[Images and Artifacts] --> Recovery
    DNS[DNS, TLS, Load Balancer, Provider Configuration] --> Recovery
    Runbooks[Runbooks and Inventory] --> Recovery

15.2. Backup Artifacts

15.3. Backup Strategies by Environment

15.4. etcd Backup with kubeadm

etcd contains Kubernetes cluster state. Snapshots contain sensitive information and must be encrypted, access-restricted, and stored outside the cluster.

# On a control-plane node of a kubeadm cluster
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd/snapshot-$(date +%F-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

sudo ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd/snapshot-*.db --write-out=table

Also back up:

sudo tar czf /var/backups/kubernetes-pki-$(date +%F).tgz /etc/kubernetes/pki
sudo tar czf /var/backups/kubernetes-manifests-$(date +%F).tgz /etc/kubernetes/manifests
sudo sha256sum /var/backups/etcd/*.db /var/backups/*.tgz > /var/backups/SHA256SUMS

15.5. Velero Concept

Velero is suitable for backups and restores of Kubernetes resources and persistent volumes. It uses an object-storage target and optionally volume snapshots or filesystem backups.

# Example: back up one namespace
velero backup create demo-$(date +%F-%H%M) --include-namespaces demo
velero backup describe demo-$(date +%F-%H%M) --details
velero backup logs demo-$(date +%F-%H%M)

# Schedule
velero schedule create demo-daily \
  --schedule="0 2 * * *" \
  --include-namespaces demo \
  --ttl 720h

# Restore
velero restore create demo-restore-$(date +%F-%H%M) --from-backup demo-daily-20260613020000
velero restore describe demo-restore-$(date +%F-%H%M) --details

15.6. CSI VolumeSnapshots

VolumeSnapshots back up volumes at the storage layer. They are fast, but depend on the CSI driver, storage backend, and location. A snapshot alone does not replace application-consistent backup.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-snapshot
  namespace: demo
spec:
  volumeSnapshotClassName: csi-snapshot-class
  source:
    persistentVolumeClaimName: data

15.7. Application-Consistent Backups

For databases, a Kubernetes volume snapshot is often not enough. You need application-specific procedures:

15.8. Backup Retention

15.9. Backup Security

• Encrypt backups at rest and in transit.
• Do not store backups only in the same cluster.
• Protect backup buckets against accidental deletion.
• Separate read and write permissions.
• Test restore with restricted permissions.
• Check whether Secrets are included in the backup.
• Document key rotation and recovery of encryption keys.

16. Restore and Disaster Recovery

16.1. DR Levels

16.2. DR Architecture

flowchart TD
    Prod[Production Cluster] --> B1[Git and IaC Mirror]
    Prod --> B2[etcd Snapshots or Managed State]
    Prod --> B3[PV Snapshots and App Backups]
    Prod --> B4[Image Registry Mirror]
    Prod --> B5[Observability Export]

    B1 --> DR[DR Cluster or New Environment]
    B2 --> DR
    B3 --> DR
    B4 --> DR
    DR --> Validate[Validation: DNS, Ingress, Data, SLO]
    Validate --> Cutover[DNS/LB Cutover]

16.3. Restore-Test Runbook

1. Select a representative workload.
2. Create or identify a valid backup.
3. Restore into an isolated namespace or test cluster.
4. Check Deployments, Pods, Services, PVCs, Secrets, and Ingress.
5. Validate application data functionally, not only Kubernetes objects.
6. Measure recovery duration.
7. Document deviations from RTO/RPO.
8. Delete test resources in a controlled way.
9. Update runbook, alerts, and backup configuration.

# Example: Velero restore into new namespace via namespace mapping
velero restore create demo-restore-test \
  --from-backup demo-daily-20260613020000 \
  --namespace-mappings demo:demo-restore-test

kubectl get all,pvc,secret,configmap -n demo-restore-test
kubectl get events -n demo-restore-test --sort-by=.lastTimestamp

16.4. etcd Restore with kubeadm as Emergency Pattern

Warning: Perform etcd restore only in a controlled environment. An incorrect restore can overwrite cluster state. Back up the current state first, even if it appears broken.

High-level process:

1. Ensure control-plane access.
2. Back up current static Pod manifests and PKI.
3. Check snapshot integrity.
4. Stop control-plane components by temporarily moving static manifests away.
5. Restore snapshot into a new etcd data directory.
6. Adjust etcd manifest to point to the new data directory.
7. Re-enable static manifests.
8. Check API server readiness.
9. Validate workload state.

# Example orientation, do not execute blindly
SNAPSHOT=/var/backups/etcd/snapshot.db
RESTORE_DIR=/var/lib/etcd-restore

sudo ETCDCTL_API=3 etcdctl snapshot status "$SNAPSHOT" --write-out=table
sudo mkdir -p "$RESTORE_DIR"
sudo etcdutl snapshot restore "$SNAPSHOT" --data-dir "$RESTORE_DIR"

# Afterwards, the etcd static pod manifest must be carefully pointed to the new data directory.
# Check /etc/kubernetes/manifests/etcd.yaml and document every change.

16.5. Managed Kubernetes DR

With managed Kubernetes, you rarely restore the provider control plane yourself. A realistic path is:

1. Create a new environment using Terraform/OpenTofu/Pulumi/provider CLI.
2. Deploy add-ons in a defined order.
3. Connect GitOps.
4. Restore Secrets/External Secrets.
5. Restore PVs and databases from snapshots/backups.
6. Check Ingress, DNS, certificates, and WAF.
7. Switch traffic in a controlled way.
8. Freeze or analyze the old environment.

16.6. DR Decision Tree

flowchart TD
    Incident[Major Outage] --> DataLoss{Data loss?}
    DataLoss -- No --> Rebuild{Cluster rebuildable?}
    Rebuild -- Yes --> GitRebuild[Rebuild from Git/IaC]
    Rebuild -- No --> NodeFix[Repair Node/Control Plane]
    DataLoss -- Yes --> BackupOk{Backup valid and tested?}
    BackupOk -- Yes --> Restore[Perform Restore]
    BackupOk -- No --> Contain[Limit damage and manually reconstruct]
    Restore --> Validate[Validate Application Functionally]
    GitRebuild --> Validate
    NodeFix --> Validate
    Validate --> Cutover[Release Traffic]
    Contain --> RCA[Root Cause Analysis]

17. Patch and Update Concept

17.1. Patch Categories

17.2. Patch Windows

17.3. Before Every Patch

kubectl config current-context
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl get pdb -A
kubectl get hpa -A
kubectl get pvc -A
velero backup get 2>/dev/null || true

Also check:

• Is there a current backup?
• Are PDBs realistic, or do they block every drain?
• Is there enough free capacity when one node is unavailable?
• Are DaemonSets tolerated and current?
• Are long-running Jobs or migrations active?
• Are critical maintenance windows, freezes, or blackout dates active?

17.4. Node Patching Baseline

NODE=<node-name>

kubectl cordon "$NODE"
kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

# Patch on the node
sudo apt update && sudo apt -y upgrade
sudo reboot

# After reboot
kubectl get node "$NODE" -o wide
kubectl uncordon "$NODE"
kubectl get pods -A -o wide --field-selector spec.nodeName="$NODE"

On RHEL/Rocky/Alma/Fedora:

sudo dnf -y update
sudo reboot

On SLES/openSUSE:

sudo zypper refresh
sudo zypper update -y
sudo reboot

17.5. Automated Node Patching

Options:

• Cluster Autoscaler plus new node images and node-pool rotation;
• Kured for reboots after Linux updates;
• cloud-native node image upgrades;
• immutable nodes with Packer, Image Builder, Talos, or Cluster API;
• GitOps-controlled maintenance jobs.

Automation is useful when it respects PDBs, capacity, maintenance windows, and alert suppression.

18. Kubernetes Upgrades

18.1. Upgrade Principles

• Do not skip minor versions unless your tooling or provider explicitly supports it.
• Upgrade control plane first, then add-ons, then nodes. Exact sequencing depends on distribution and provider.
• Check version-skew rules for API server, kubelet, kube-proxy, controller, scheduler, and kubectl.
• Check removed or deprecated API versions before upgrading.
• Check CNI, CSI, Ingress, cert-manager, OPA/Gatekeeper/Kyverno, External Secrets, Argo CD/Flux, and monitoring components.
• Plan node drain and capacity.
• Keep a rollforward plan. Rollback for Kubernetes minor upgrades is often not trivial.

18.2. Upgrade Order as Diagram

flowchart TD
    A[Check Release Notes and Version Skew] --> B[Test Staging Upgrade]
    B --> C[Check Backups and Restore Point]
    C --> D[Check API Deprecations]
    D --> E[Check Add-on Compatibility]
    E --> F[Upgrade Control Plane]
    F --> G[Upgrade Core Add-ons]
    G --> H[Upgrade Node Pools]
    H --> I[Validate Workloads and SLOs]
    I --> J[Update Git/IaC/Runbooks]

18.3. Check API Deprecations

kubectl api-resources
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis || true
kubectl get events -A --sort-by=.lastTimestamp | grep -i deprecated || true

# With pluto or kubent, if installed
pluto detect-files -d ./k8s
kubent

18.4. kubeadm Upgrade Pattern

Simplified example pattern for a minor upgrade:

# On first control-plane node
sudo apt-mark unhold kubeadm
sudo apt update
sudo apt install -y kubeadm=1.36.1-1.1
sudo apt-mark hold kubeadm

kubeadm version
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.36.1

sudo apt-mark unhold kubelet kubectl
sudo apt install -y kubelet=1.36.1-1.1 kubectl=1.36.1-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Then on additional control-plane nodes and workers, sequentially:
sudo kubeadm upgrade node

Drain and update a node:

NODE=<node-name>
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=10m
# On the node, update kubeadm/kubelet/kubectl
sudo kubeadm upgrade node
sudo systemctl restart kubelet
kubectl uncordon "$NODE"

18.5. EKS Upgrade Pattern

CLUSTER=demo-eks
REGION=eu-central-1
TARGET=1.36

aws eks describe-cluster --name "$CLUSTER" --region "$REGION" \
  --query 'cluster.{name:name,version:version,status:status,endpoint:endpoint}'

# Start control-plane upgrade
aws eks update-cluster-version \
  --name "$CLUSTER" \
  --region "$REGION" \
  --kubernetes-version "$TARGET"

# Observe status
aws eks describe-update --name "$CLUSTER" --region "$REGION" --update-id <update-id>

# Update kubeconfig
aws eks update-kubeconfig --name "$CLUSTER" --region "$REGION"

# Upgrade managed node group
eksctl upgrade nodegroup \
  --cluster "$CLUSTER" \
  --name <node-group-name> \
  --region "$REGION"

After EKS upgrades, check:

• EKS add-ons such as VPC CNI, CoreDNS, kube-proxy, EBS CSI;
• IAM Roles for Service Accounts or Pod Identity;
• Security Groups, Load Balancer Controller;
• Managed Node Groups, Fargate Profiles;
• CloudWatch/AMP/ADOT configuration.

18.6. AKS Upgrade Pattern

RG=rg-kubernetes
CLUSTER=demo-aks
TARGET=1.36.1

az aks show -g "$RG" -n "$CLUSTER" --query '{name:name,kubernetesVersion:kubernetesVersion,powerState:powerState.code}'
az aks get-upgrades -g "$RG" -n "$CLUSTER" -o table

# Upgrade control plane and node pools
az aks upgrade \
  -g "$RG" \
  -n "$CLUSTER" \
  --kubernetes-version "$TARGET" \
  --yes

# Update kubeconfig
az aks get-credentials -g "$RG" -n "$CLUSTER" --overwrite-existing

# Upgrade node images separately, if planned
az aks nodepool upgrade \
  -g "$RG" \
  --cluster-name "$CLUSTER" \
  -n <nodepool> \
  --node-image-only

After AKS upgrades, check:

• Azure CNI or Kubenet/Azure CNI Overlay;
• CSI Driver for Disk/File;
• Azure Monitor Agent;
• Managed Identity and Workload Identity;
• Ingress Controller, Application Gateway Ingress Controller, or Gateway components;
• Planned Maintenance and Auto-Upgrade settings.

18.7. GKE Upgrade Pattern

PROJECT=demo-project
REGION=europe-west3
CLUSTER=demo-gke
TARGET=1.36.1-gke.100

gcloud container clusters describe "$CLUSTER" \
  --region "$REGION" \
  --project "$PROJECT" \
  --format='value(currentMasterVersion,currentNodeVersion,status)'

# Control plane
gcloud container clusters upgrade "$CLUSTER" \
  --region "$REGION" \
  --project "$PROJECT" \
  --master \
  --cluster-version "$TARGET"

# Node pool
gcloud container clusters upgrade "$CLUSTER" \
  --region "$REGION" \
  --project "$PROJECT" \
  --node-pool <nodepool> \
  --cluster-version "$TARGET"

gcloud container clusters get-credentials "$CLUSTER" --region "$REGION" --project "$PROJECT"

After GKE upgrades, check:

• Release Channel and Maintenance Windows;
• Node Auto-Upgrades and Surge Settings;
• GKE Dataplane, Ingress/Gateway Controller;
• Cloud Logging/Monitoring;
• Workload Identity;
• CSI Driver and Backup for GKE.

18.8. Update kind, minikube, and k3d

Local clusters are often better recreated than live-upgraded.

# kind
kind get clusters
kind export logs --name demo ./kind-logs-demo || true
kind delete cluster --name demo
kind create cluster --name demo --image kindest/node:v1.36.1

# minikube
minikube profile list
minikube update-check
minikube delete -p demo
minikube start -p demo --kubernetes-version=v1.36.1 --driver=docker

# k3d
k3d cluster list
k3d cluster delete demo
k3d cluster create demo --image rancher/k3s:v1.36.1-k3s1 -p "8080:80@loadbalancer"

18.9. k3s, RKE2, k0s, and Talos

Always document:

• target version;
• supported upgrade paths;
• etcd/control-plane backup;
• add-on compatibility;
• drain strategy;
• validation after each node.

19. Add-on and Operator Lifecycle

19.1. Critical Add-ons

19.2. Add-on Upgrade Runbook

1. Read release notes.
2. Identify CRD changes.
3. Back up CRDs and custom resources.
4. Check webhooks and admission policies.
5. Perform staging upgrade.
6. Check metrics and logs.
7. Use production maintenance window.
8. After upgrade, check rollout, webhooks, CRDs, and Events.

# Back up CRDs
kubectl get crd -o yaml > crds-before.yaml
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o yaml > webhooks-before.yaml

# Check Helm release
helm list -A
helm history <release> -n <namespace>
helm template <release> <chart> -n <namespace> -f values.yaml > rendered.yaml
kubectl apply --dry-run=server -f rendered.yaml

# Upgrade
helm upgrade --install <release> <chart> -n <namespace> -f values.yaml
kubectl rollout status deployment/<controller> -n <namespace> --timeout=180s

19.3. CRD Risks

CRDs are API extensions. Faulty CRD or webhook updates can block deployments throughout the cluster. Check especially:

• conversion webhooks;
• removed fields;
• incompatible custom resources;
• validation rules;
• finalizers;
• controller versions;
• backup/restore capability for custom resources.

20. Node and Operating System Maintenance

20.1. Node Maintenance with Drain

sequenceDiagram
    participant Op as Operator
    participant API as Kubernetes API
    participant Node as Node
    participant Sched as Scheduler
    participant Pods as Pods

    Op->>API: cordon node
    API->>Node: do not schedule new Pods
    Op->>API: drain node
    API->>Pods: evict Pods according to PDB
    Sched->>API: schedule replacement Pods on other Nodes
    Op->>Node: patch or reboot OS/Runtime
    Op->>API: uncordon node
    Sched->>Node: allow new Pods again

20.2. Drain Pitfalls

20.3. Reboot Runbook

NODE=<node-name>

kubectl get node "$NODE" -o wide
kubectl describe node "$NODE" | egrep 'Taints|Conditions|Capacity|Allocatable' -A5
kubectl get pods -A -o wide --field-selector spec.nodeName="$NODE"

kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=15m

# Reboot node
ssh "$NODE" 'sudo reboot'

# Wait and validate
kubectl get node "$NODE" -w
kubectl uncordon "$NODE"
kubectl get pods -A -o wide --field-selector spec.nodeName="$NODE"

20.4. Immutable Node Rotation

In cloud and Cluster API environments, node replacement is often cleaner than in-place patching:

1. Build new node image.
2. Create new Node Pool / MachineDeployment version.
3. Add a small number of nodes.
4. Drain workloads.
5. Remove old nodes.
6. Check capacity and cost.

This model reduces configuration drift and makes rollback through old node pools easier.

21. Certificates, Secrets, and Key Material

21.1. Certificate Inventory

21.2. Check kubeadm Certificates

sudo kubeadm certs check-expiration
sudo kubeadm certs renew --help

21.3. Check cert-manager

kubectl get certificate,certificaterequest,order,challenge -A
kubectl describe certificate <name> -n <namespace>
kubectl logs -n cert-manager deploy/cert-manager --tail=100

21.4. Secrets Operations

• Base64 in Kubernetes Secrets is not encryption.
• Enable Encryption at Rest for Kubernetes Secrets where you operate the control plane.
• Use external secret systems for production Secrets: Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, External Secrets Operator, SOPS.
• Avoid Secrets in Helm values, pipeline logs, and support bundles.
• Rotations must reload or restart workloads.
• Check RBAC: who can read Secrets?

kubectl get secrets -A
kubectl auth can-i get secrets -n demo
kubectl auth can-i get secrets -n demo --as system:serviceaccount:demo:demo-api

21.5. Secret Rotation Runbook

1. Create new Secret in the secret system.
2. Update Kubernetes Secret or let ExternalSecret sync.
3. Trigger workload restart or hot reload.
4. Validate functionality.
5. Disable old Secret.
6. Check audit and access.

kubectl rollout restart deployment/demo-api -n demo
kubectl rollout status deployment/demo-api -n demo --timeout=180s

22. Storage Operations

22.1. Storage Operations Questions

• Which StorageClass is default?
• Are Retain/Delete reclaim policies defined?
• Does the CSI driver support snapshots?
• Are volumes bound to zones?
• How is volume expansion performed?
• Are backups application-consistent?
• How do you detect full PVCs?
• How are PVs protected when namespaces are deleted?

22.2. Storage Triage

kubectl get storageclass
kubectl get pvc -A
kubectl get pv
kubectl get volumesnapshotclass 2>/dev/null || true
kubectl get volumesnapshot -A 2>/dev/null || true
kubectl get volumeattachment -A 2>/dev/null || true
kubectl describe pvc <pvc> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp | tail -80

22.3. PVC Full

NS=demo
POD=$(kubectl get pod -n $NS -l app.kubernetes.io/name=demo-api -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n $NS "$POD" -- df -h
kubectl describe pvc data -n $NS

Possible actions:

• clean up application data;
• reduce retention;
• expand PVC if the StorageClass allows it;
• migrate data to a new volume;
• expand storage backend;
• create backup before dangerous changes.

22.4. Expand PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: demo
spec:
  resources:
    requests:
      storage: 100Gi

Then check:

kubectl get pvc data -n demo -w
kubectl describe pvc data -n demo

22.5. StatefulSet Maintenance

For StatefulSets, consider:

• stable Pod names;
• stable PVC mapping;
• ordered rollouts;
• data replication at application level;
• backup before schema or version changes;
• clear sequence for primaries/replicas.

23. Network, DNS, and Ingress Operations

23.1. Network Layers

flowchart TD
    Client[Client / Browser / API Consumer] --> DNS[DNS]
    DNS --> LB[Load Balancer]
    LB --> Ingress[Ingress Controller or Gateway]
    Ingress --> Service[Kubernetes Service]
    Service --> EndpointSlice[EndpointSlices]
    EndpointSlice --> Pod[Pod IP]
    Pod --> CNI[CNI Dataplane]
    Pod --> Egress[Egress, NAT, Firewall, Proxy]

23.2. Network Triage

NS=demo
kubectl get svc,endpoints,endpointslices -n $NS
kubectl get ingress -A
kubectl get gateway,httproute -A 2>/dev/null || true
kubectl get networkpolicy -A
kubectl get pods -n kube-system -o wide

# DNS test
kubectl run net-debug -n $NS --rm -it --image=busybox:1.36 --restart=Never -- sh
# inside the Pod:
# nslookup kubernetes.default.svc.cluster.local
# nslookup demo-api.demo.svc.cluster.local
# wget -S -O- http://demo-api.demo.svc.cluster.local

23.3. Service Without Endpoints

Causes:

• Service selector does not match Pod labels;
• Pods are not Ready;
• wrong namespace;
• EndpointSlices are not created;
• port name or targetPort is wrong.

kubectl get svc demo-api -n demo -o yaml
kubectl get pods -n demo --show-labels
kubectl get endpoints,endpointslices -n demo -l kubernetes.io/service-name=demo-api
kubectl describe pod -n demo -l app.kubernetes.io/name=demo-api

23.4. Ingress/Gateway Operations

Check:

• DNS points to correct Load Balancer;
• Load Balancer is provisioned;
• IngressClass or GatewayClass is correct;
• certificate is valid;
• Service has Endpoints;
• controller logs show no routing error;
• NetworkPolicies allow traffic.

kubectl get ingress -A
kubectl describe ingress demo-api -n demo
kubectl get ingressclass
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100 2>/dev/null || true

kubectl get gatewayclass,gateway,httproute -A 2>/dev/null || true
kubectl describe httproute demo-api -n demo 2>/dev/null || true

23.5. WSL and Windows Hosts File in Operations

When you run Kubernetes locally in WSL, Edge runs on Windows, not Linux. For browser-friendly local names, you may need to maintain the Windows hosts file:

# PowerShell as Administrator
notepad C:\Windows\System32\drivers\etc\hosts

From WSL, the file is usually at:

sudo cp /mnt/c/Windows/System32/drivers/etc/hosts /mnt/c/Windows/System32/drivers/etc/hosts.bak.$(date +%F-%H%M%S)
# Editing requires Windows administrator rights. If it fails, use PowerShell as Administrator.

Example entries:

127.0.0.1 demo.local
127.0.0.1 grafana.local
127.0.0.1 prometheus.local

Operational rule: hosts file changes are local configuration. Document them in the lab runbook, but do not treat them as production DNS.

24. Security Operations

24.1. Security Operations Layers

flowchart TD
    Identity[Identity and Access] --> RBAC[RBAC Least Privilege]
    RBAC --> Admission[Admission Policies]
    Admission --> PodSecurity[Pod Security Standards]
    PodSecurity --> Runtime[Runtime Security]
    Runtime --> Network[NetworkPolicies and Egress]
    Network --> Secrets[Secrets and Keys]
    Secrets --> Audit[Audit Logs]
    Audit --> Response[Incident Response]

24.2. Recurring Security Checks

24.3. RBAC Review

kubectl get clusterrolebinding -o wide
kubectl get rolebinding -A -o wide
kubectl get serviceaccount -A
kubectl auth can-i '*' '*' --as system:serviceaccount:demo:demo-api -n demo
kubectl auth can-i get secrets --as system:serviceaccount:demo:demo-api -n demo

24.4. Pod Security Review

kubectl get ns --show-labels | grep pod-security || true
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" privileged="}{range .spec.containers[*]}{.securityContext.privileged}{" "}{end}{"\n"}{end}'

24.5. Incident Response for Secret Leak

1. Confirm the leak, but do not spread the Secret further.
2. Rotate or disable the affected access immediately.
3. Update workloads with the new Secret.
4. Check audit logs.
5. Clean persistent artifacts: Git, CI logs, tickets, chat, support bundles.
6. Fix the cause: RBAC, logging, pipeline masking, Secret scanner.
7. Document postmortem.

25. Capacity, Performance, and Cost

25.1. Capacity Dimensions

25.2. Check Requests and Limits

kubectl top pods -A --containers 2>/dev/null || true
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{" req.cpu="}{.resources.requests.cpu}{" req.mem="}{.resources.requests.memory}{" lim.cpu="}{.resources.limits.cpu}{" lim.mem="}{.resources.limits.memory}{"; "}{end}{"\n"}{end}'

25.3. Cost Traps

• Services of type LoadBalancer that nobody uses anymore.
• PVCs/PVs with Retain left behind after namespace deletion.
• Snapshots without lifecycle.
• Logging retention too long or debug logs too noisy.
• Oversized node pools.
• HPA without Cluster Autoscaler or with overly high requests.
• NAT Gateway, data transfer, and cross-zone traffic in clouds.

25.4. Capacity Review

kubectl get nodes -o wide
kubectl describe nodes | egrep 'Name:|Allocatable:|Allocated resources:' -A10
kubectl get hpa -A
kubectl get pdb -A
kubectl get pvc -A
kubectl get svc -A --field-selector spec.type=LoadBalancer 2>/dev/null || true

26. GitOps and Release Operations

26.1. GitOps Operating Model

flowchart LR
    Dev[Change in Git] --> PR[Pull Request]
    PR --> Checks[CI, Tests, Policy, Security]
    Checks --> Merge[Merge]
    Merge --> GitOps[Argo CD or Flux]
    GitOps --> K8s[Kubernetes API]
    K8s --> Status[Status, Events, Health]
    Status --> GitOps
    GitOps --> Alert[Drift or Sync Error]

26.2. GitOps Rules

• Production is not changed directly with kubectl edit.
• Hotfixes in the cluster are immediately reflected back into Git.
• Every application has an owner, health checks, and sync strategy.
• Secrets are not stored in Git in plaintext.
• CRDs and operators have their own rollout sequence.
• Automatic sync is not appropriate for every resource.
• Prune is dangerous when ownership is not clean.

26.3. Argo CD Operations

argocd app list
argocd app get demo-api
argocd app diff demo-api
argocd app sync demo-api
argocd app wait demo-api --health --timeout 180
kubectl get applications -n argocd
kubectl logs -n argocd deploy/argocd-application-controller --tail=100

26.4. Flux Operations

flux get sources all -A
flux get kustomizations -A
flux get helmreleases -A
flux reconcile source git <source-name> -n flux-system
flux reconcile kustomization <name> -n flux-system
kubectl logs -n flux-system deploy/kustomize-controller --tail=100

26.5. Release Gates

Before production:

• build is reproducible;
• image is signed or at least scanned;
• manifests are server-side validated;
• policy checks passed;
• rollout in stage succeeded;
• SLO dashboard is ready;
• rollback was tested;
• DB migration evaluated with return path;
• backup or snapshot exists;
• communication window is clear.

27. Environment-Specific Operations

27.1. WSL2 on Windows 11

Day-2 focus areas:

• check WSL version, distribution, and Docker integration;
• understand Windows/WSL networking and port mapping;
• maintain Windows hosts file for Edge access;
• recreate local clusters regularly;
• do not store production Secrets in the lab environment;
• control memory usage of Docker Desktop and WSL.

wsl -l -v
wsl --status
wsl --shutdown

kubectl config get-contexts
docker ps
kind get clusters
minikube profile list
k3d cluster list

27.2. kind

Operating approach:

• treat clusters as short-lived;
• version the configuration in kind-config.yaml;
• load images locally;
• document port mappings;
• do not treat test data as durable.

kind get clusters
kind export logs --name demo ./kind-logs-demo
kind load docker-image ghcr.io/example/demo-api:1.2.3 --name demo
kubectl cluster-info --context kind-demo

27.3. minikube

minikube profile list
minikube status -p demo
minikube addons list -p demo
minikube service list -p demo
minikube tunnel -p demo
minikube logs -p demo

Watch for:

• running minikube tunnel processes;
• driver changes;
• local resource limits;
• add-on versions;
• profile names.

27.4. k3d

k3d cluster list
k3d node list
k3d kubeconfig get demo
k3d cluster delete demo

k3d is suitable for fast k3s labs. For persistent data, explicitly mount and back up volumes.

27.5. Local VM with VirtualBox/VMware/Vagrant

Day-2 focus areas:

• VM snapshots before risky changes;
• document host-only/NAT/bridged networking;
• check time/NTP;
• monitor disk usage;
• check kubelet and runtime as system services;
• version Vagrantfile and provisioning scripts.

vagrant status
vagrant snapshot list
vagrant snapshot save cp1 before-upgrade-$(date +%F)
vagrant ssh cp1

27.6. Remote VM

Also check:

• SSH access and emergency access;
• provider firewall and local firewall;
• fixed IPs and DNS;
• provider snapshots;
• offsite backup;
• costs for running VMs, disks, and snapshots.

ssh <user>@<node>
sudo ufw status verbose 2>/dev/null || true
sudo firewall-cmd --list-all 2>/dev/null || true

27.7. Bare Metal

Day-2 focus areas:

• hardware monitoring;
• firmware and BIOS/BMC versions;
• RAID/disk health;
• local and external storage backups;
• spare parts and node capacity;
• physical access and out-of-band management;
• clear drain and replacement processes.

sudo smartctl -a /dev/sda 2>/dev/null || true
sudo ipmitool sensor 2>/dev/null || true
sudo journalctl -p warning -n 200 --no-pager

27.8. kubeadm

You are responsible for:

• control plane;
• etcd;
• certificates;
• kubelet;
• container runtime;
• CNI/CSI;
• API and node upgrades;
• backups and restore.

sudo kubeadm certs check-expiration
sudo kubeadm upgrade plan
sudo crictl ps | grep kube
kubectl get --raw='/readyz?verbose'

27.9. Managed Kubernetes: EKS, AKS, and GKE

Day-2 focus areas:

• provider versions and support windows;
• node pools and node images;
• managed add-ons;
• cloud IAM and Workload Identity;
• cloud Load Balancers and DNS;
• cloud logging/monitoring;
• backup and DR for workloads;
• cost control.

# EKS
aws eks list-clusters
aws eks describe-cluster --name <cluster> --region <region>
aws eks list-addons --cluster-name <cluster> --region <region>

# AKS
az aks list -o table
az aks show -g <rg> -n <cluster>
az aks get-upgrades -g <rg> -n <cluster> -o table

# GKE
gcloud container clusters list
gcloud container clusters describe <cluster> --region <region>
gcloud container operations list --region <region>

27.10. Talos

Talos strongly separates OS management from classic SSH operations. Operations run through talosctl and MachineConfig.

talosctl health
talosctl dashboard
talosctl service kubelet
talosctl logs kubelet
talosctl get members

27.11. OpenShift

OpenShift includes integrated cluster operators, Security Context Constraints, built-in monitoring, and its own upgrade flow.

oc get clusterversion
oc get clusteroperators
oc adm upgrade
oc get co
oc get nodes

28. Runbook Catalog for Common Incidents

This section provides compact, reusable runbooks. Adapt commands, namespaces, and tooling to your environment.

28.1. Pod Pending

Symptom: Pods are not scheduled.

Common causes:

• insufficient CPU or memory;
• NodeSelector/Affinity mismatch;
• taint without toleration;
• PVC Pending;
• ImagePull is not yet relevant because the Pod has not started.

First commands:

kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe nodes
kubectl get pvc -n <namespace>

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.2. CrashLoopBackOff

Symptom: Container starts, exits, and is restarted repeatedly.

Common causes:

• application error;
• missing configuration;
• Secret missing;
• probe too aggressive;
• migration fails.

First commands:

kubectl logs <pod> -n <namespace> --previous
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.3. ImagePullBackOff

Symptom: Kubernetes cannot pull the container image.

Common causes:

• wrong image name;
• tag missing;
• registry unreachable;
• ImagePullSecret wrong;
• rate limit.

First commands:

kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.4. Service Without Endpoints

Symptom: Service does not route to Pods.

Common causes:

• selector mismatch;
• Pods not Ready;
• wrong port;
• wrong namespace.

First commands:

kubectl get svc,endpoints,endpointslices -n <namespace>
kubectl get pods -n <namespace> --show-labels
kubectl describe svc <service> -n <namespace>

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.5. Cluster DNS Broken

Symptom: Service names cannot be resolved.

Common causes:

• CoreDNS not ready;
• NetworkPolicy blocks DNS;
• upstream DNS broken;
• NodeLocal DNS Cache problem.

First commands:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl run dns-debug --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.6. Ingress Does Not Route

Symptom: External HTTP/HTTPS traffic does not reach the Service.

Common causes:

• DNS wrong;
• Load Balancer not ready;
• IngressClass wrong;
• Service without Endpoints;
• TLS Secret missing;
• controller error.

First commands:

kubectl get ingress -A
kubectl describe ingress <name> -n <namespace>
kubectl logs -n <ingress-namespace> deploy/<controller> --tail=100
curl -vk https://<host>

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.7. Gateway API Route Not Accepted

Symptom: HTTPRoute or Gateway does not take effect.

Common causes:

• GatewayClass missing;
• listener mismatch;
• Route not attached;
• AllowedRoutes blocks it;
• ReferenceGrant missing.

First commands:

kubectl get gatewayclass,gateway,httproute -A
kubectl describe httproute <route> -n <namespace>
kubectl describe gateway <gateway> -n <namespace>

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.8. PVC Pending

Symptom: PersistentVolumeClaim is not bound.

Common causes:

• StorageClass missing;
• Provisioner broken;
• zone/topology mismatch;
• quota exceeded;
• CSI Controller faulty.

First commands:

kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get pods -n kube-system | grep csi

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.9. Node NotReady

Symptom: Node is not Ready.

Common causes:

• kubelet broken;
• CNI broken;
• runtime broken;
• DiskPressure;
• MemoryPressure;
• network disconnected.

First commands:

kubectl describe node <node>
ssh <node> 'sudo systemctl status kubelet --no-pager'
ssh <node> 'sudo journalctl -u kubelet -n 200 --no-pager'
ssh <node> 'sudo crictl ps -a'

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.10. API Server Slow or Unreachable

Symptom: Kubernetes API responds slowly or not at all.

Common causes:

• etcd slow;
• API load;
• webhook hanging;
• network problem;
• control-plane resources exhausted.

First commands:

kubectl get --raw='/readyz?verbose'
kubectl get --raw='/metrics' | grep apiserver_request
sudo crictl logs $(sudo crictl ps --name kube-apiserver -q | head -1)

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.11. Admission Webhook Blocks Deployments

Symptom: Apply or rollout fails because of webhook errors.

Common causes:

• Webhook Service unreachable;
• TLS certificate expired;
• failurePolicy: Fail;
• operator broken.

First commands:

kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get svc -A | grep webhook
kubectl get certificates -A
kubectl get events -A --sort-by=.lastTimestamp | grep -i webhook

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.12. HPA Does Not Scale

Symptom: Horizontal Pod Autoscaler does not react.

Common causes:

• metrics-server missing;
• requests missing;
• metric unavailable;
• maxReplicas reached.

First commands:

kubectl get hpa -A
kubectl describe hpa <hpa> -n <namespace>
kubectl top pods -n <namespace>
kubectl get apiservice | grep metrics

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.13. OOMKilled

Symptom: Container is terminated because of memory limit.

Common causes:

• memory leak;
• limit too low;
• requests/limits wrong;
• Node MemoryPressure.

First commands:

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl top pod <pod> -n <namespace> --containers

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.14. CPU Throttling

Symptom: Container is slowed by CPU limits.

Common causes:

• limit too low;
• load spikes;
• unfavorable threading;
• missing requests.

First commands:

Prometheus: container_cpu_cfs_throttled_periods_total
kubectl top pod <pod> -n <namespace> --containers
kubectl describe pod <pod> -n <namespace>

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.15. Backup Failed

Symptom: Backup job or Velero backup failed.

Common causes:

• object storage unreachable;
• permission missing;
• VolumeSnapshotClass missing;
• timeout;
• too much data.

First commands:

velero backup get
velero backup describe <backup> --details
velero backup logs <backup>
kubectl logs -n velero deploy/velero --tail=200

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.16. Restore Incomplete

Symptom: Objects or data are missing after restore.

Common causes:

• namespace mapping wrong;
• CRDs missing;
• StorageClass missing;
• Secrets missing;
• PV snapshot not restored.

First commands:

velero restore get
velero restore describe <restore> --details
kubectl get all,pvc,secret,configmap -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.17. GitOps OutOfSync

Symptom: GitOps reports drift or sync error.

Common causes:

• manual change in cluster;
• webhook blocks;
• CRD missing;
• prune dangerous;
• Helm template error.

First commands:

argocd app get <app>
argocd app diff <app>
flux get kustomizations -A
kubectl get events -A --sort-by=.lastTimestamp | tail -100

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.18. Certificate Expires

Symptom: TLS or Kubernetes certificate is nearing expiry.

Common causes:

• cert-manager renewal broken;
• Issuer unreachable;
• kubeadm certs not renewed;
• ACME challenge failed.

First commands:

kubectl get certificates -A
kubectl describe certificate <cert> -n <namespace>
sudo kubeadm certs check-expiration

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.19. NetworkPolicy Blocks Traffic

Symptom: Connections between Pods or to DNS do not work.

Common causes:

• default deny without allow;
• DNS egress missing;
• labels do not match;
• CNI does not support policy.

First commands:

kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <policy> -n <namespace>
kubectl run net-debug --rm -it --image=busybox:1.36 --restart=Never -- sh

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

28.20. Log Ingestion Too Expensive or Too Noisy

Symptom: Logging backend grows heavily or causes high cost.

Common causes:

• debug logs enabled;
• too many labels;
• no retention;
• no exclusions;
• access logs unfiltered.

First commands:

Check backend ingestion metrics
kubectl logs --tail
Check Prometheus/Loki/cloud cost dashboard

Decision:

• If the cause points to configuration, correct Git/manifest and roll out again.
• If the cause points to infrastructure, escalate to platform, cloud, or storage/network owner.
• If data loss is possible, stop experiments and check backup/restore path.
• Document cause, impact, fix, and alert/runbook improvement.

29. Checklists

29.1. Production Readiness Checklist

29.2. Backup Checklist

• Backup target is outside the cluster.
• Backup target is encrypted.
• Access to backup is separate from cluster-admin access.
• Retention is defined.
• Restore was tested.
• Backups generate alerts on failures.
• Secrets and key material are considered.
• Databases have application-consistent backups.
• Backup costs are monitored.

29.3. Upgrade Checklist

• Target version is supported.
• Version skew is checked.
• Release notes are read.
• API deprecations are checked.
• Add-on compatibility is checked.
• Staging upgrade succeeded.
• Backup and restore point exist.
• Maintenance window is communicated.
• PDBs and capacity are checked.
• Rollback/rollforward is documented.
• Post-check is defined.

29.4. Incident Checklist

• Set severity.
• Determine affected services and customer impact.
• Capture current state: Events, logs, metrics, screenshots, relevant manifests.
• No destructive commands without backup/return path.
• Inform owner and on-call.
• Perform mitigation.
• Measure stability.
• Create postmortem.
• Improve alerts, runbooks, or architecture.

30. Workshop Labs

30.1. Lab: Cluster Triage in 10 Minutes

Objective: You identify basic cluster problems systematically.

Tasks:

1. Check context and nodes.
2. List non-running Pods.
3. Sort Events by time.
4. Check kube-system.
5. Check resources with kubectl top.
6. Document three findings and their priority.

30.2. Lab: Fix CrashLoopBackOff

1. Roll out a faulty Pod.
2. Check describe, logs, and Events.
3. Correct ConfigMap or start command.
4. Roll out again.
5. Add an alert and a runbook.

30.3. Lab: Backup and Restore with Velero

1. Install Velero in a lab cluster.
2. Create namespace, Deployment, Service, and PVC.
3. Create a backup.
4. Delete the namespace.
5. Restore it.
6. Functionally verify that the data is correct.

30.4. Lab: Node Drain and Maintenance

1. Create a Deployment with three replicas and a PDB.
2. Drain a node.
3. Check whether the workload remains available.
4. Simulate reboot or patch.
5. Uncordon the node.
6. Adjust PDB and resources if drain is blocked.

30.5. Lab: Kubernetes Upgrade Rehearsal

1. Create a non-production cluster.
2. Install monitoring and an example add-on.
3. Check API deprecations.
4. Perform a patch or minor upgrade.
5. Validate workloads, DNS, Ingress, storage, and alerts.
6. Document duration, errors, and improvements.

30.6. Lab: Improve Alerting

1. Install Prometheus/Alertmanager.
2. Create a CrashLoop.
3. Write a PrometheusRule.
4. Add runbook annotation.
5. Check whether alert, dashboard, and runbook fit together.

30.7. Lab: DR Exercise

1. Define RTO/RPO for a demo app.
2. Back up Git state, Kubernetes resources, and PVC.
3. Restore into a new namespace or cluster.
4. Measure time until functionality.
5. Evaluate deviation from RTO/RPO.

31. Appendices: Commands, Manifests, and Scripts

31.1. Day-2 Healthcheck Script

Save the following script as k8s-day2-healthcheck.sh and run it against the desired context.

#!/usr/bin/env bash
set -euo pipefail

NS_FILTER="${1:-}"
TS="$(date +%F-%H%M%S)"
OUT="k8s-healthcheck-$TS"
mkdir -p "$OUT"

run() {
  local name="$1"
  shift
  echo "==> $name"
  ("$@" > "$OUT/$name.txt" 2>&1) || true
}

run context kubectl config current-context
run version kubectl version
run nodes kubectl get nodes -o wide
run pods-all kubectl get pods -A -o wide
run events kubectl get events -A --sort-by=.lastTimestamp
run namespaces kubectl get ns --show-labels
run pdb kubectl get pdb -A
run hpa kubectl get hpa -A
run pvc kubectl get pvc -A
run storageclass kubectl get storageclass
run ingress kubectl get ingress -A
run networkpolicy kubectl get networkpolicy -A
run apiservices kubectl get apiservice
run readyz kubectl get --raw=/readyz?verbose
run livez kubectl get --raw=/livez?verbose
run top-nodes kubectl top nodes
run top-pods kubectl top pods -A

if command -v velero >/dev/null 2>&1; then
  run velero-backups velero backup get
  run velero-schedules velero schedule get
fi

if [ -n "$NS_FILTER" ]; then
  run namespace-all kubectl get all -n "$NS_FILTER" -o wide
  run namespace-events kubectl get events -n "$NS_FILTER" --sort-by=.lastTimestamp
fi

tar czf "$OUT.tgz" "$OUT"
echo "Healthcheck written: $OUT.tgz"

31.2. etcd Backup Script for kubeadm

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="${BACKUP_DIR:-/var/backups/etcd}"
RETENTION_DAYS="${RETENTION_DAYS:-14}"
TS="$(date +%F-%H%M%S)"
SNAPSHOT="$BACKUP_DIR/etcd-snapshot-$TS.db"

sudo mkdir -p "$BACKUP_DIR"
sudo ETCDCTL_API=3 etcdctl snapshot save "$SNAPSHOT" \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

sudo ETCDCTL_API=3 etcdctl snapshot status "$SNAPSHOT" --write-out=table
sudo sha256sum "$SNAPSHOT" | sudo tee "$SNAPSHOT.sha256" >/dev/null
sudo find "$BACKUP_DIR" -type f -name 'etcd-snapshot-*.db' -mtime +"$RETENTION_DAYS" -print -delete

echo "etcd snapshot created: $SNAPSHOT"

31.3. Velero Schedule Example

velero schedule create cluster-daily \
  --schedule="0 2 * * *" \
  --ttl 720h

velero schedule create critical-hourly \
  --schedule="0 * * * *" \
  --include-namespaces critical-apps \
  --ttl 168h

velero backup get
velero schedule get

31.4. NetworkPolicy for DNS and App Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-ingress-to-app
  namespace: demo
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: demo-api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

31.5. Final Check After Maintenance

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl get deploy,sts,ds -A
kubectl get hpa,pdb -A
kubectl get pvc -A
kubectl get ingress -A
kubectl get certificates -A 2>/dev/null || true
kubectl get --raw='/readyz?verbose' 2>/dev/null || true

Then assess:

• Are there new alerts?
• Are SLO dashboards stable?
• Are backups still successful after maintenance?
• Are GitOps applications synchronized?
• Did costs or resource usage increase unexpectedly?
• Does a runbook need to be adjusted?

32. Sources and Further Documentation

This material is written as a workshop and operations reference. For real production environments, also check the current vendor and project documentation.

• Kubernetes documentation: cluster administration, observability, logging, metrics, probes, kubeadm upgrade, version skew, and etcd operations: https://kubernetes.io/docs/
• Kubernetes release information: https://kubernetes.io/releases/
• etcd disaster recovery: https://etcd.io/docs/
• Velero documentation: https://velero.io/docs/
• Prometheus documentation: https://prometheus.io/docs/
• Alertmanager documentation: https://prometheus.io/docs/alerting/latest/alertmanager/
• OpenTelemetry documentation: https://opentelemetry.io/docs/
• AWS EKS documentation: https://docs.aws.amazon.com/eks/
• Azure AKS documentation: https://learn.microsoft.com/azure/aks/
• Google GKE documentation: https://cloud.google.com/kubernetes-engine/docs
• Grafana Loki documentation: https://grafana.com/docs/loki/
• Argo CD documentation: https://argo-cd.readthedocs.io/
• Flux documentation: https://fluxcd.io/flux/
• cert-manager documentation: https://cert-manager.io/docs/
• External Secrets Operator documentation: https://external-secrets.io/

Closing Note

Day-2 Operations are a continuous control loop. Good Kubernetes operations do not come from a single tool, but from clear ownership, verifiable backups, tested restores, controlled upgrades, good observability, and clear runbooks. Use local environments such as WSL, kind, minikube, and k3d to practice these procedures safely. Then transfer them deliberately and adapt them to VM, bare-metal, and cloud clusters.