1. Goal and Scope

This document is a complete troubleshooting reference for Kubernetes. Use it when a cluster, node, add-on, or application does not behave as expected. It is deliberately more than a list of individual failure patterns: it describes working models, environment-specific aspects, runbooks, diagnostic commands, evidence preservation, and validation.

1.1. Target Audiences

1.2. Non-Goals

This document does not replace product-specific support contracts, a formal incident-response policy, or provider status pages. It also does not replace understanding your platform architecture. It does provide a complete working frame so that you do not execute individual commands without a plarrives.

1.3. Guiding Idea

A Kubernetes failure is almost never only a Kubernetes failure. It usually sits at an interface:

• Manifest against API schema,
• desired state against actual state,
• Scheduler against node capacity,
• Pod against container runtime,
• Service against labels and endpoints,
• DNS against CoreDNS and network,
• Ingress or Gateway against controller and load balancer,
• PVC against StorageClass and CSI,
• RBAC against identity,
• add-on against CRDs, webhooks, and versions,
• cloud controller against IAM, quotas, subnets, or security groups.

2. Troubleshooting Principles

2.1. Check from the Outside In

1. User path: Is the problem reproducible from the user perspective?
2. DNS and routing: Does the client reach the correct name, IP, and port?
3. Ingress, Gateway, or Load Balancer: Does traffic arrive at the cluster?
4. Service: Does the Service have Endpoints or EndpointSlices?
5. Pod: Is the Pod Running and Ready?
6. Container: Does the container start reliably and listen on the expected port?
7. Application: Are configuration, Secrets, dependencies, and data access correct?
8. Node: Does the node have runtime, CNI, storage, resource, or OS issues?
9. Control Plane: Do the API Server, controllers, scheduler, and etcd work?
10. Provider: Do cloud IAM, quotas, subnets, firewalls, routes, tags, or limits block the path?

2.2. Observe First, Then Change

Do not change a manifest immediately just because a Pod fails. First collect status, events, logs, recent changes, and environment context. Without the initial state, you cannot later prove whether your change helped or only moved the failure.

2.3. Events Are Often More Important Than Status

kubectl get shows the current state. kubectl describe and kubectl get events show how Kubernetes reached that state. For scheduling, image pull, mount, probe, admission, and controller problems, events are often the fastest source of evidence.

2.4. Separate Symptom from Cause

2.5. Reproduce Minimally

If you do not know whether the problem is in the app or the platform, deploy a minimal reference app and a debug Pod. If the reference app works, the issue is likely in the app, manifest, image, Secret, or policy. If it does not work, check the platform path, network, DNS, runtime, and add-ons.

3. Initial Triage in Ten Minutes

Use this block when you quickly need to decide whether to continue with workload, node, control plane, network, storage, or provider diagnostics.

# 1. Context check
kubectl config current-context
kubectl cluster-info
kubectl version

# 2. Clusterzustand grob check
kubectl get nodes -o wide
kubectl get pods -A -o wide --sort-by=.metadata.creationTimestamp
kubectl get events -A --sort-by=.lastTimestamp | tail -100

# 3. Workload and Namespace check
NS=demo
APP=demo
kubectl get all -n "$NS"
kubectl get deploy,rs,sts,ds,job,cronjob -n "$NS"
kubectl get pod -n "$NS" -o wide --show-labels

# 4. errorhafte Pods identifizieren
kubectl get pods -A \
  --field-selector=status.phase!=Running \
  -o wide

# 5. Details, Logs and letzte Logs check
POD=<pod>
kubectl describe pod "$POD" -n "$NS"
kubectl logs "$POD" -n "$NS" --all-containers --tail=200
kubectl logs "$POD" -n "$NS" --all-containers --previous --tail=200

# 6. networkpfad check
kubectl get svc,endpoints,endpointslices,ingress -n "$NS" -o wide
kubectl get networkpolicy -A

# 7. Storagepfad check
kubectl get storageclass
kubectl get pv,pvc -A

# 8. RBAC check
kubectl auth can-i get pods -n "$NS"
kubectl auth can-i '*' '*' -n "$NS"

3.1. Immediate Classification

4. Decision Tree

flowchart TD
    A[Problem gemeldet] --> B{kubectl API reachable?}
    B -- Nein --> C[Context, kubeconfig, VPN, API Server, Provider check]
    B -- Ja --> D{Betrifft es many Namespaces?}
    D -- Ja --> E{Nodes gesund?}
    E -- Nein --> F[Node, Runtime, CNI, Disk, certificates, Provider]
    E -- Ja --> G{CoreDNS, CNI, CSI or Admission auffällig?}
    G -- Ja --> H[Add-on Runbook]
    G -- Nein --> I[Control Plane, Quotas, Cloud Limits, globale Policies]
    D -- Nein --> J{Pod present?}
    J -- Nein --> K[Deployment, ReplicaSet, Job, GitOps, Helm check]
    J -- Ja --> L{Pod Ready?}
    L -- Nein --> M[Scheduling, ImagePull, CrashLoop, Probes, Mounts]
    L -- Ja --> N{Service has Endpoints?}
    N -- Nein --> O[Labels, Selector, Readiness, EndpointSlices]
    N -- Ja --> P{Traffic reaches Service?}
    P -- Nein --> Q[DNS, NetworkPolicy, kube-proxy/eBPF, CNI]
    P -- Ja --> R[App, Backend, databank, external Dependency]

4.1. Data Flow for HTTP Issues

sequenceDiagram
    participant Browser as Browser or Client
    participant DNS as DNS
    participant LB as Load Balancer
    participant GW as Ingress/Gateway Controller
    participant SVC as Kubernetes Service
    participant EPS as EndpointSlice
    participant Pod as Pod
    participant App as App Container

    Browser->>DNS: Hostname auflösen
    DNS-->>Browser: IP-Adresse
    Browser->>LB: TCP/TLS/HTTP Request
    LB->>GW: Weiterleitung an Controller
    GW->>SVC: Routing after Host/Path/Route
    SVC->>EPS: Backend-Auswahl
    EPS->>Pod: Pod-IP and Port
    Pod->>App: Container-Port
    App-->>Browser: response or error

4.2. Rule for Fast Scoping

• HTTP 404 deutet oft on wrong Ingress-/Gateway-Route or wrong Host hin.
• HTTP 503 deutet oft on missing or not bereite Backends hin.
• Connection refused often indicates the wrong port, no running listener, or a node/port-mapping problem.
• Timeout deutet oft on Firewall, NetworkPolicy, Route, CNI, Load Balancer or provider issue hin.
• TLS error often indicates certificate, SNI, Secret, trust store, or controller-configuration issues.

5. Environment Matrix

6. Diagnostic Tools

6.1. Kubernetes CLI

kubectl version
kubectl cluster-info
kubectl config get-contexts
kubectl api-resources
kubectl api-versions
kubectl explain deployment.spec.template.spec.containers
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'
kubectl get --raw='/metrics' | head

6.2. Workload Diagnosis

kubectl get pod -A -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers --tail=200
kubectl logs <pod> -n <namespace> --all-containers --previous --tail=200
kubectl exec -it <pod> -n <namespace> -- sh
kubectl debug -it <pod> -n <namespace> --image=nicolaka/netshoot --target=<container>

6.3. Node Diagnosis

kubectl describe node <node>
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp
kubectl top nodes
kubectl top pods -A
ssh <node>
systemctl status kubelet containerd crio
journalctl -u kubelet -n 300 --no-pager
journalctl -u containerd -n 300 --no-pager
crictl info
crictl ps -a
crictl pods
crictl logs <container-id>

6.4. Network Diagnosis

kubectl get svc,endpoints,endpointslices -A -o wide
kubectl get ingress -A
kubectl get gateway,httproute -A
kubectl get networkpolicy -A
kubectl run netshoot -n default --rm -it --restart=Never --image=nicolaka/netshoot -- bash
kubectl run dnsutils -n default --rm -it --restart=Never --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- nslookup kubernetes.default

6.5. Storage Diagnosis

kubectl get storageclass
kubectl get pv,pvc -A -o wide
kubectl describe pvc <pvc> -n <namespace>
kubectl get volumeattachment
kubectl get csidriver
kubectl get pods -A | grep -i csi

6.6. Security Diagnosis

kubectl auth whoami
kubectl auth can-i get pods -n <namespace>
kubectl auth can-i '*' '*' --as system:serviceaccount:<namespace>:<serviceaccount>
kubectl get role,rolebinding -n <namespace>
kubectl get clusterrole,clusterrolebinding
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding

6.7. Linux and Network Tools

ip addr
ip route
ip neigh
ss -tulpn
ss -antp
resolvectl status || cat /etc/resolv.conf
nft list ruleset || iptables-save
journalctl -xe --no-pager
dmesg -T | tail -200
df -h
findmnt
free -m
uptime

6.8. Cloud CLI

# AWS EKS
aws eks describe-cluster --name <cluster> --region <region>
aws eks update-kubeconfig --name <cluster> --region <region>
aws eks list-addons --cluster-name <cluster>
aws ec2 describe-security-groups --group-ids <sg-id>

# Azure AKS
az aks show -g <resource-group> -n <cluster>
az aks get-credentials -g <resource-group> -n <cluster> --overwrite-existing
az aks nodepool list -g <resource-group> --cluster-name <cluster>
az network nsg rule list -g <resource-group> --nsg-name <nsg>

# Google GKE
gcloud container clusters describe <cluster> --region <region>
gcloud container clusters get-credentials <cluster> --region <region>
gcloud container operations list --region <region>
gcloud compute firewall-rules list

7. Standard Data Collection

7.1. Minimal Support Bundle Set

Collect this information before making a change:

TS=$(date -u +%Y%m%dT%H%M%SZ)
OUT="k8s-debug-$TS"
mkdir -p "$OUT"

kubectl version -o yaml > "$OUT/version.yaml" 2>&1 || true
kubectl cluster-info > "$OUT/cluster-info.txt" 2>&1 || true
kubectl config current-context > "$OUT/current-context.txt" 2>&1 || true
kubectl get nodes -o wide > "$OUT/nodes.txt" 2>&1 || true
kubectl describe nodes > "$OUT/nodes-describe.txt" 2>&1 || true
kubectl get pods -A -o wide > "$OUT/pods-wide.txt" 2>&1 || true
kubectl get events -A --sort-by=.lastTimestamp > "$OUT/events.txt" 2>&1 || true
kubectl get ns -o yaml > "$OUT/namespaces.yaml" 2>&1 || true
kubectl get svc,endpoints,endpointslices -A -o wide > "$OUT/network.txt" 2>&1 || true
kubectl get ingress -A -o yaml > "$OUT/ingress.yaml" 2>&1 || true
kubectl get gateway,httproute -A -o yaml > "$OUT/gateway-api.yaml" 2>&1 || true
kubectl get storageclass,pv,pvc -A -o yaml > "$OUT/storage.yaml" 2>&1 || true
kubectl get networkpolicy -A -o yaml > "$OUT/networkpolicies.yaml" 2>&1 || true
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o yaml > "$OUT/webhooks.yaml" 2>&1 || true
kubectl get crd > "$OUT/crds.txt" 2>&1 || true

tar -czf "$OUT.tar.gz" "$OUT"
echo "$OUT.tar.gz"

7.2. Namespace-Specific Collection

NS=<namespace>
TS=$(date -u +%Y%m%dT%H%M%SZ)
OUT="k8s-debug-$NS-$TS"
mkdir -p "$OUT"

kubectl get all -n "$NS" -o wide > "$OUT/all-wide.txt" 2>&1 || true
kubectl get all -n "$NS" -o yaml > "$OUT/all.yaml" 2>&1 || true
kubectl get events -n "$NS" --sort-by=.lastTimestamp > "$OUT/events.txt" 2>&1 || true
kubectl get configmap,secret,sa,role,rolebinding -n "$NS" -o yaml > "$OUT/config-security.yaml" 2>&1 || true
kubectl get pvc -n "$NS" -o yaml > "$OUT/pvc.yaml" 2>&1 || true
kubectl get ingress,gateway,httproute,networkpolicy -n "$NS" -o yaml > "$OUT/networking.yaml" 2>&1 || true

for pod in $(kubectl get pod -n "$NS" -o name | sed 's#pod/##'); do
  kubectl describe pod "$pod" -n "$NS" > "$OUT/pod-$pod-describe.txt" 2>&1 || true
  kubectl logs "$pod" -n "$NS" --all-containers --tail=500 > "$OUT/pod-$pod-logs.txt" 2>&1 || true
  kubectl logs "$pod" -n "$NS" --all-containers --previous --tail=500 > "$OUT/pod-$pod-logs-previous.txt" 2>&1 || true
done

tar -czf "$OUT.tar.gz" "$OUT"

7.3. What You Should Not Collect Uncontrolled

• Secrets in the Klartext,
• production Tokens,
• completee kubeconfigs with Client-certificatesn,
• personenbezogene data from Logs,
• databankdumps,
• TLS-Private-Keys,
• customersnamen and internal hostnames, if the Bundle external shared is.

7.4. Redaction Before Sharing

# Vorsicht: always an a Kopie arbeiten.
cp -r k8s-debug-<timestamp> k8s-debug-redacted
find k8s-debug-redacted -type f -name '*.yaml' -o -name '*.txt' | while read -r f; do
  sed -i \
    -e 's/[A-Za-z0-9_\.-]*@[A-Za-z0-9_\.-]*/REDACTED_EMAIL/g' \
    -e 's/password:.*/password: REDACTED/g' \
    -e 's/token:.*/token: REDACTED/g' \
    -e 's/client-key-data:.*/client-key-data: REDACTED/g' \
    "$f"
done

8. Security and Evidence Preservation

8.1. When a Security Incident Is Possible

When you suspect a security incident, you do not only stabilize technically. You preserve evidence and minimize changes to the compromised system.

1. Incident-ID create.
2. time, Melder, Symptome and Scope dokumentieren.
3. Betropene Namespaces, Pods, Nodes, ServiceAccounts and Images erfassen.
4. Logs, Events, Audit-Logs and Container-Metadaten save.
5. networkaccess begrenzen, without Beweise zu vernichten.
6. Tokens and Secrets rotieren, if Missbrauch wahrscheinlich is.
7. Roll out the workload again from a trusted source.
8. Close policy gap.

8.2. No Blind Cleanup

Do not immediately run kubectl delete pod on a suspicious Pod if forensic information matters. First collect:

NS=<namespace>
POD=<pod>
kubectl get pod "$POD" -n "$NS" -o yaml > incident-pod.yaml
kubectl describe pod "$POD" -n "$NS" > incident-pod-describe.txt
kubectl logs "$POD" -n "$NS" --all-containers --timestamps --tail=-1 > incident-pod-logs.txt
kubectl logs "$POD" -n "$NS" --all-containers --previous --timestamps --tail=-1 > incident-pod-logs-previous.txt
kubectl get events -n "$NS" --sort-by=.lastTimestamp > incident-events.txt
kubectl get sa,role,rolebinding -n "$NS" -o yaml > incident-rbac.yaml

8.3. Isolate Instead of Deleting

During active abuse, you can apply a restrictive NetworkPolicy, remove the Service selector, or set the Deployment replica count to zero. Which action is appropriate depends on whether availability or evidence preservation matters more.

# Quick Isolation eines Namespace, if your CNI NetworkPolicies durchsetzt.
kubectl apply -n <namespace> -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: emergency-default-deny
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
EOF

9. API, kubeconfig, and kubectl Issues

9.1. kubectl Uses the Wrong Context

Symptoms

• resources missing angeblich.
• You are working in the wrong Cluster.
• kubectl shows old Clusterinformationen.

Diagnosis

kubectl config get-contexts
kubectl config current-context
kubectl config view --minify
kubectl cluster-info

Actions

kubectl config use-context <context>
kubectl config set-context --current --namespace=<namespace>

Validation

kubectl get ns
kubectl auth whoami

9.2. API Server Not Reachable

Symptoms

• Unable to connect to the server.
• i/o timeout.
• connection refused.
• x509: certificate signed by unknown authority.

Diagnosis

kubectl cluster-info
kubectl get --raw='/readyz?verbose'
kubectl config view --minify --raw
curl -k https://<api-server>:6443/readyz
nc -vz <api-server> 6443

Bei kubeadm-Control-Plane:

sudo crictl ps -a | grep kube-apiserver
sudo crictl logs <apiserver-container-id>
sudo journalctl -u kubelet -n 300 --no-pager
sudo ls -l /etc/kubernetes/manifests/

Common Causes

• wrong kubeconfig,
• VPN or Firewall blocks,
• API Server Pod starts not,
• certificate expired,
• Load-Balancer-Endpoint wrong,
• cloud control plane unavailable,
• local kind/minikube/k3d-Container stopped.

9.3. API Is Slow

Diagnosis

kubectl get --raw='/readyz?verbose'
kubectl get --raw='/metrics' | grep -E 'apiserver_request_duration|apiserver_current_inflight_requests' | head -50
kubectl get events -A --sort-by=.lastTimestamp | tail -100

Causes

• too many Watcher,
• langsame or blockierende Admission Webhooks,
• etcd-Latenz,
• API-Priority-and-Fairness-Drosselung,
• large CRDs or controllers with too many requeues,
• Cloud-API-Rate-Limits.

9.4. kubectl apply Fails Because of Schema or Admission

kubectl apply --server-side --dry-run=server -f manifest.yaml
kubectl diff -f manifest.yaml
kubectl explain <kind>.spec
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get events -A --sort-by=.lastTimestamp | tail -50

If the error message comes from a webhook, check the webhook Service:

kubectl get validatingwebhookconfiguration <name> -o yaml
kubectl get svc,endpoints -n <webhook-namespace>
kubectl logs -n <webhook-namespace> deploy/<webhook-deployment>

10. Workload Troubleshooting

10.1. Understand Pod States

10.2. Read Logs Correctly

# Aktuelle Logs
kubectl logs <pod> -n <namespace> --all-containers --tail=200

# Logs the beforeigen container starts
kubectl logs <pod> -n <namespace> --all-containers --previous --tail=200

# Logs eines Deployments
kubectl logs deploy/<deployment> -n <namespace> --all-containers --tail=200

# Logs with Zeitstempeln
kubectl logs <pod> -n <namespace> --timestamps --all-containers

10.3. Exit Codes

10.4. Container Starts Locally but Not in Kubernetes

Check Unterschiede zwischen localm Docker-Run and Kubernetes:

• Environment Variables,
• Working Directory,
• Entrypoint and Args,
• User and file permissions,
• read-only Root Filesystem,
• Capabilities,
• Secrets and ConfigMaps,
• Service DNS,
• networkpolicy,
• resourceslimits,
• time zone and Locale,
• Architektur the Images (amd64, arm64).

kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl get pod <pod> -n <namespace> -o yaml | sed -n '/containers:/,/volumes:/p'
kubectl exec -it <pod> -n <namespace> -- env | sort
kubectl exec -it <pod> -n <namespace> -- id
kubectl exec -it <pod> -n <namespace> -- pwd
kubectl exec -it <pod> -n <namespace> -- ls -la

10.5. Ephemeral Debug Container

If an image contains no shell tool, use kubectl debug:

kubectl debug -it pod/<pod> -n <namespace> \
  --image=nicolaka/netshoot \
  --target=<container> \
  -- bash

Not every cluster allows ephemeral containers. Admission policies, Pod Security, RBAC, or older Kubernetes versions can block this.

11. Scheduling, resource, and Autoscaling Issues

11.1. Read Scheduler Events

kubectl describe pod <pod> -n <namespace> | sed -n '/Events:/,$p'
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i -E 'FailedScheduling|Insufficient|taint|affinity|preempt'

11.2. Check resource Situation

kubectl top nodes
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory
kubectl describe node <node> | sed -n '/Allocated resources:/,/Events:/p'
kubectl get resourcequota,limitrange -A

11.3. Common Scheduling Causes

11.4. HPA Does Not Scale

kubectl get hpa -A
kubectl describe hpa <hpa> -n <namespace>
kubectl top pod -n <namespace>
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
kubectl logs -n kube-system deploy/metrics-server --tail=200

Causes:

• Metrics Server missing or is not ready,
• Pods haben no CPU-Requests,
• HPA-Metrik is wrong referenziert,
• Custom Metrics Adapter missing,
• stabilizationWindowSeconds delays scaling,
• Cluster Autoscaler cannot add nodes,
• PodDisruptionBudget or quota blocks indirectly.

11.5. Cluster Autoscaler or Node Autoscaling Does Not Respond

Check:

kubectl -n kube-system get deploy | grep -i autoscaler
kubectl -n kube-system logs deploy/cluster-autoscaler --tail=300
kubectl get pods -A | grep Pending
kubectl describe pod <pending-pod> -n <namespace>

Cloud-specific:

• AWS: Auto Scaling Group, Launch Template, IAM, subnets, tags.
• Azure: VMSS, Quotas, Node-Pool-Modus, Zonen.
• GCP: Instance Group, Quotas, Reservations, Autopilot-Constraints.

12. Network, DNS, Service, Ingress, and Gateway Issues

12.1. Check the Network Path Systematically

NS=<namespace>
APP=<app-label>
SVC=<service>

kubectl get pod -n "$NS" -l app="$APP" -o wide --show-labels
kubectl get svc "$SVC" -n "$NS" -o wide
kubectl get endpoints "$SVC" -n "$NS" -o yaml
kubectl get endpointslices -n "$NS" -l kubernetes.io/service-name="$SVC" -o yaml
kubectl get networkpolicy -n "$NS" -o yaml

12.2. Service without Endpoints

A Service does not route to Pods if its selector finds no matching and ready Pods.

kubectl get svc <service> -n <namespace> -o yaml | yq '.spec.selector'
kubectl get pods -n <namespace> --show-labels
kubectl get endpoints <service> -n <namespace>
kubectl describe pod <pod> -n <namespace> | grep -A20 Conditions

Check:

• Labels passen exakt zu Selector?
• Pods are Ready?
• Service port and targetPort korrekt?
• Named Port exists in the Pod?
• EndpointSlices are through Controller creates?

12.3. Check DNS

kubectl run dnsutils -n default --rm -it --restart=Never \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 \
  -- nslookup kubernetes.default

kubectl run netshoot -n default --rm -it --restart=Never \
  --image=nicolaka/netshoot -- bash

# Im Debug-Pod:
cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local
nslookup <service>.<namespace>.svc.cluster.local
dig <service>.<namespace>.svc.cluster.local

CoreCheck DNS:

kubectl -n kube-system get deploy,po,svc,endpoints -l k8s-app=kube-dns
kubectl -n kube-system logs deploy/coredns --tail=200
kubectl -n kube-system get configmap coredns -o yaml

12.4. NetworkPolicy Blocks Traffic

First check whether your CNI enforces NetworkPolicies at all. Without a supporting CNI, a NetworkPolicy is only an API object without enforcement.

kubectl get networkpolicy -A
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get pod -n <namespace> --show-labels

Typische error:

• Default-Deny without DNS-Egress-Ausnahme,
• wrong podSelector,
• wrong namespaceSelector,
• Labels missing,
• Egress zu external APIs missing,
• Service Mesh mTLS and NetworkPolicy conflict with each other,
• Policy was in wrongm Namespace angelegt.

12.5. Ingress Does Not Work

kubectl get ingress -A
kubectl describe ingress <ingress> -n <namespace>
kubectl get ingressclass
kubectl get svc,endpoints -n <namespace>
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=300
curl -v -H 'Host: <host>' http://<ingress-ip>/<path>

Check:

• Ingress Controller runs?
• ingressClassName matches?
• DNS shows on correct IP?
• Host and Path passen to the request?
• Service port points on existing Port?
• TLS Secret liegt in the selben Namespace wie Ingress?
• Load Balancer or NodePort is reachable?

12.6. Gateway API Does Not Work

kubectl get gatewayclass,gateway,httproute,grpcroutes,tlsroute -A
kubectl describe gateway <gateway> -n <namespace>
kubectl describe httproute <route> -n <namespace>
kubectl get events -A --sort-by=.lastTimestamp | grep -i -E 'gateway|route|listener|accepted|resolved'

Check Conditions:

• Accepted,
• Programmed,
• ResolvedRefs,
• Listener-Konflikte,
• wrong parentRefs,
• Cross-Namespace-Referenzen without ReferenceGrant,
• GatewayClass without Controller.

12.7. MTU and Fragmentation Issues

Symptoms:

• small requests work, large requests hang,
• TLS Handshake bricht ab,
• Service Mesh shows sporadische Timeouts,
• only specific Node-zu-Node-paths affected.

Diagnosis:

ip link
ping -M do -s 1472 <ziel-ip>
tracepath <ziel-ip>
kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash

Check Overlay-CNI, VPN, Cloud-VPC, WireGuard, VXLAN, Geneve, IPsec and Provider-Limits.

13. Storage, CSI, and Data Issues

13.1. PVC remains Pending

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass -o yaml
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Causes:

• no default StorageClass,
• wrong storageClassName,
• dynamic provisioning unavailable,
• CSI Controller runs not,
• cloud quota exhausted,
• Zone/topology does not match,
• AccessMode not supported,
• VolumeBindingMode WaitForFirstConsumer waits on Scheduling.

13.2. Pod Stuck in ContainerCreating Because of Mount

kubectl describe pod <pod> -n <namespace> | sed -n '/Events:/,$p'
kubectl describe pvc <pvc> -n <namespace>
kubectl get pv <pv> -o yaml
kubectl get volumeattachment
kubectl get pods -A | grep -i csi

Look for events such as:

• FailedMount,
• Unable to attach or mount volumes,
• Multi-Attach error,
• timed out waiting for the condition,
• permission denied,
• no such file or directory.

13.3. Multi-Attach Error

An RWO volume often cannot be mounted read-write on multiple nodes at the same time.

kubectl get pod -n <namespace> -o wide | grep <app>
kubectl get volumeattachment | grep <pv>
kubectl describe volumeattachment <name>

Actions:

• Workload kontrolliert on a Node verschieben,
• check hanging attachments,
• StatefulSet instead of Deployment use,
• AccessMode and StorageBackend check,
• in an emergency, use provider-specific detach only after checking data consistency.

13.4. Data Is Gone after Pod Restart

Check, whether you really persistence nutzt:

kubectl get pod <pod> -n <namespace> -o yaml | yq '.spec.volumes'
kubectl get pvc -n <namespace>

Typical causes:

• emptyDir instead of PVC,
• local paths in Container-Filesystem,
• StatefulSet without volumeClaimTemplates,
• Helm values disable persistence,
• wrong namespace checked,
• Restore in wrong Cluster erfolgt.

13.5. CSI Controller or Node Plugin Defective

kubectl get csidriver
kubectl get pods -A | grep -i csi
kubectl logs -n <csi-namespace> deploy/<csi-controller> --tail=300
kubectl logs -n <csi-namespace> ds/<csi-node> --tail=300
kubectl get events -A --sort-by=.lastTimestamp | grep -i csi

Check Cloud-IAM, Secret for Storage-Backend, Node-Permissions, Kernel-Module, Multipath, NFS-Erreichbarkeit, iSCSI, Ceph-Monitore, Longhorn-Engine, Rook-Ceph-status or Provider-Quota.

14. Node, Runtime, and Operating System Issues

14.1. Node NotReady

kubectl get nodes -o wide
kubectl describe node <node>
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp

Auf the Node:

systemctl status kubelet --no-pager
journalctl -u kubelet -n 300 --no-pager
systemctl status containerd crio docker --no-pager
journalctl -u containerd -n 300 --no-pager
crictl info
crictl ps -a
ip addr
ip route
df -h
free -m

14.2. Node Conditions

14.3. container runtime Defective

sudo crictl info
sudo crictl ps -a
sudo crictl images
sudo crictl pods
sudo journalctl -u containerd -n 300 --no-pager
sudo journalctl -u crio -n 300 --no-pager

Actions:

• Runtime-Service new start, if sicher vertretbar,
• clean disk,
• resolve image pull problems,
• Registry-Mirror check,
• cgroup-Treiber-Konsistenz zwischen kubelet and Runtime check,
• Kernel- and OverlayFS-issues check.

14.4. Resolve DiskPressure

df -h
df -ih
sudo you -xh /var/lib/containerd /var/log /var/lib/kubelet 2>/dev/null | sort -h | tail -40
sudo crictl images
sudo crictl ps -a

Sichere Actions:

# Only ungenutzte Images delete; before Impact check.
sudo crictl rmi --prune
sudo journalctl --vacuum-time=7d

Do not delete blindly:

• /var/lib/kubelet/pods,
• dataverzeichnisse from CSI/Longhorn/Rook,
• etcd-dataverzeichnis,
• production App-data.

14.5. CNI Defective on the Node

Symptoms:

• Pods remain in ContainerCreating,
• FailedCreatePodSandBox,
• Node NetworkUnavailable,
• Pod-to-Pod traffic disrupted,
• DNS-Time-outs.

kubectl get pods -A -o wide | grep -i -E 'calico|cilium|flannel|weave|antrea|canal'
ls -l /etc/cni/net.d/
ls -l /opt/cni/bin/
journalctl -u kubelet -n 300 --no-pager | grep -i cni

14.6. Time and NTP Issues

Zeitversatz verursacht TLS-, Token-, Lease-, etcd- and TTL-issues.

timedatectl
chronyc tracking || true
journalctl -u systemd-timesyncd -n 100 --no-pager || true

Symptoms:

• certificates appear not yet valid or expired,
• Nodes ernewrn Leases not cleanly,
• Jobs are unerwaits bereinigt,
• etcd elections accumulate,
• OIDC tokens are rejected.

15. Control Plane, etcd, and kubeadm Issues

15.1. Check kubeadm Control Plane

sudo ls -l /etc/kubernetes/manifests
sudo crictl ps -a | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'
sudo crictl logs <container-id>
sudo journalctl -u kubelet -n 500 --no-pager
kubectl get componentstatuses 2>/dev/null || true
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'

15.2. Static Pods Do Not Start

Static Pods are read by kubelet from /etc/kubernetes/manifests. If the API Server or etcd does not start, check:

• YAML-Syntax,
• certificatespfade,
• HostPath-Mounts,
• file- and directoryrechte,
• etcd-dataverzeichnis,
• Port-Konflikte,
• Node-time,
• container runtime.

15.3. etcd Is Not Healthy

export ETCDCTL_API=3
sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

Check:

• Quorum,
• Disk-Latenz,
• available inodes,
• certificates,
• network zwischen Control-Plane-Nodes,
• Snapshots and dataverzeichnis,
• Uhrzeit.

15.4. Certificates Expired

sudo kubeadm certs check-expiration
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates

Actions for kubeadm:

sudo kubeadm certs renew all
sudo systemctl restart kubelet

Then check kubeconfigs and redistribute them if necessary:

sudo kubeadm kubeconfig user --client-name kubernetes-admin > admin.conf

15.5. kubeadm Upgrade Failed

kubeadm upgrade plan
kubectl get nodes
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get update
sudo apt-get install -y kubeadm=<version>
sudo kubeadm upgrade apply <version>
sudo apt-get install -y kubelet=<version> kubectl=<version>
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon <node>

Important:

• Do not skip minor versions.
• Erst Control Plane, dann Worker Nodes.
• Vorher etcd-Backup create.
• Check CNI, CSI, and add-on compatibility.
• Version-Skew beachten.

15.6. Scheduler Does Not Schedule

kubectl -n kube-system get pod -l component=kube-scheduler -o wide
kubectl -n kube-system logs -l component=kube-scheduler --tail=300
kubectl get events -A --sort-by=.lastTimestamp | grep -i scheduling

Bei statischem kubeadm-Scheduler:

sudo crictl ps -a | grep kube-scheduler
sudo crictl logs <scheduler-container-id>
sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml

16. Add-on, Operator, Helm, and GitOps Issues

16.1. Basic Add-on Check

kubectl get pods -A | grep -E 'coredns|metrics-server|ingress|gateway|cert-manager|external-dns|prometheus|grafana|loki|argocd|flux|calico|cilium|flannel|rook|longhorn|velero'
kubectl get crd | grep -E 'cert-manager|gateway|prometheus|argoproj|fluxcd|kyverno|cilium|rook|longhorn|velero'
kubectl get events -A --sort-by=.lastTimestamp | tail -200

16.2. CRD Missing or Version Mismatch

Symptoms:

• no matches for kind,
• Helm install fails,
• Controller verarbeitet resources not,
• GitOps reports dry-run failed.

Diagnosis:

kubectl api-resources | grep -i <kind>
kubectl get crd | grep -i <domain>
kubectl explain <kind> --api-version=<group>/<version>

16.3. Helm Release Failed

helm list -A
helm status <release> -n <namespace>
helm history <release> -n <namespace>
helm get values <release> -n <namespace>
helm get manifest <release> -n <namespace>
helm template <release> <chart> -n <namespace> -f values.yaml > rendered.yaml
kubectl apply --dry-run=server -f rendered.yaml

Actions:

helm rollback <release> <revision> -n <namespace>
helm upgrade --install <release> <chart> -n <namespace> -f values.yaml --atomic --timeout 10m

16.4. Argo CD Sync or Health Faulty

kubectl -n argocd get applications.argoproj.io
kubectl -n argocd describe application <app>
kubectl -n argocd logs deploy/argocd-application-controller --tail=300
kubectl -n argocd logs deploy/argocd-repo-server --tail=300
argocd app get <app>
argocd app diff <app>
argocd app sync <app> --dry-run

Check:

• Repo-access,
• Pfad and Branch,
• Helm/Kustomize-values,
• CRD-Reihenfolge,
• Namespace-Erstellung,
• Sync Waves,
• resource Hooks,
• RBAC of the Argo CD controller identity,
• Drift through manuelle changes.

16.5. Flux Reconciliation Faulty

flux get sources all -A
flux get kustomizations -A
flux get helmreleases -A
flux logs --all-namespaces --tail=200
kubectl -n flux-system logs deploy/source-controller --tail=200
kubectl -n flux-system logs deploy/kustomize-controller --tail=200
kubectl -n flux-system logs deploy/helm-controller --tail=200

16.6. cert-manager Does Not Issue a Certificate

kubectl get certificate,certificaterequest,order,challenge -A
kubectl describe certificate <cert> -n <namespace>
kubectl describe challenge <challenge> -n <namespace>
kubectl logs -n cert-manager deploy/cert-manager --tail=300

Check:

• Issuer/ClusterIssuer Ready,
• DNS-01/HTTP-01-Erreichbarkeit,
• IngressClass,
• ACME-Rate-Limits,
• Secret-Namespace,
• Clock-Skew,
• external DNS-Zone and Credentials.

17. Security, RBAC, Admission, and Secret Issues

17.1. Debug Forbidden Systematically

kubectl auth whoami
kubectl auth can-i <verb> <resource> -n <namespace>
kubectl auth can-i <verb> <resource> -n <namespace> --as <user-or-serviceaccount>
kubectl get role,rolebinding -n <namespace> -o yaml
kubectl get clusterrole,clusterrolebinding -o yaml

ServiceAccount-Format:

kubectl auth can-i get pods -n demo --as system:serviceaccount:demo:app

Check:

• Role exists?
• Does the RoleBinding bind the correct identity?
• Namespace stimmt?
• Is a ClusterRoleBinding needed, or is it too broad?
• resource liegt in API-Gruppe?
• Subresource required, etwa pods/log, pods/exec, deployments/scale?

17.2. Admission Webhook Blocks Resources

kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl describe validatingwebhookconfiguration <name>
kubectl get svc,endpoints -n <webhook-namespace>
kubectl logs -n <webhook-namespace> deploy/<webhook-controller> --tail=300
kubectl get events -A --sort-by=.lastTimestamp | grep -i webhook

Typische error:

• Webhook-Service has no Endpoints,
• CA Bundle wrong,
• Timeout zu niedrig,
• failurePolicy: Fail blocks for Ausfall,
• Webhook matcht too many resources,
• NetworkPolicy blocks API Server to the Webhook,
• certificate the Webhooks expired.

17.3. Pod Security Admission Blocks Pod

kubectl get ns <namespace> --show-labels
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i 'podsecurity\|restricted\|baseline'

Check Felder wie:

• runAsNonRoot,
• allowPrivilegeEscalation,
• capabilities.drop,
• seccompProfile,
• hostNetwork, hostPID, hostIPC,
• privileged,
• hostPath,
• readOnlyRootFilesystem.

17.4. Secret Missing or Wrong Key

kubectl get secret -n <namespace>
kubectl describe pod <pod> -n <namespace> | grep -A20 -i secret
kubectl get secret <secret> -n <namespace> -o jsonpath='{.data}'

failure patterns:

• CreateContainerConfigError,
• secret not found,
• couldn't find key,
• App starts, but reports authentication errors,
• Helm has Secret in anderem Namespace created,
• External Secrets Operator has Sync-error.

17.5. Image Signature or Policy Blocks Rollout

Check Kyverno, Gatekeeper, Sigstore/Cosign-Verifikation or eigene Admission-Controller:

kubectl get clusterpolicy,policy -A 2>/dev/null || true
kubectl get constrainttemplates,constraints -A 2>/dev/null || true
kubectl get events -A --sort-by=.lastTimestamp | grep -i -E 'policy|admission|cosign|signature|verify'

18. Observability, Monitoring, Logging, and Alerting Issues

18.1. Metrics Missing

kubectl top nodes
kubectl top pods -A
kubectl get apiservice | grep metrics
kubectl describe apiservice v1beta1.metrics.k8s.io
kubectl -n kube-system logs deploy/metrics-server --tail=300

Causes:

• Metrics Server missing,
• kubelet TLS or Auth blocks,
• Nodes not reachable,
• aggregated APIService unavailable,
• Firewall blocks Port 10250,
• certificate matches not,
• Managed-Provider has eigenes Monitoring anders configured.

18.2. Prometheus Does Not Scrape Targets

kubectl -n monitoring get pod,svc,servicemonitor,podmonitor,prometheusrule
kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090

Check in Prometheus:

• status > Targets,
• ServiceMonitor-Label-Selector,
• Namespace-Selector,
• Port-Name in the Service,
• TLS- and Auth-Einstellungen,
• NetworkPolicy,
• RBAC for Prometheus.

18.3. Grafana Shows No Data

kubectl -n monitoring get pod,svc,secret,configmap | grep -i grafana
kubectl -n monitoring logs deploy/grafana --tail=200

Check:

• Datasource-URL,
• Service-Name,
• Auth Secret,
• TLS Trust,
• Dashboards ConfigMaps,
• Zeitbereich in the Dashboard,
• PromQL-Query.

18.4. Logs Missing

kubectl get pods -A | grep -i -E 'fluent|promtail|alloy|vector|otel|loki|elastic|opensearch'
kubectl logs -n <logging-namespace> ds/<agent> --tail=300
kubectl logs -n <logging-namespace> deploy/<backend> --tail=300

Check:

• schreibt the App on stdout/stderr?
• Node-Agent runs on allen Nodes?
• Log-Pfad the Runtime matches?
• RBAC the Agents reicht?
• Backend reachable?
• Tenant, Label, Index or Stream wrong?
• Does retention delete earlier than expected?
• Cloud Logging API or Agent deaktiviert?

18.5. Alerts Do Not Fire

kubectl -n monitoring get prometheusrule,alertmanagerconfig
kubectl -n monitoring logs deploy/alertmanager-main --tail=200 2>/dev/null || true
kubectl -n monitoring logs statefulset/alertmanager-main --tail=200 2>/dev/null || true

Check:

• PrometheusRule is geladen?
• Does the expression return data?
• for-Dauer still not expired?
• Alertmanager-Routing matches?
• Silence aktiv?
• Does inhibition suppress the alert?
• Receiver-Secret korrekt?
• Egress zu Slack, Teams, PagerDuty or SMTP erlaubt?

19. Backup, Restore, and Disaster Recovery Issues

19.1. Check Backup Capability

Kubernetes-Backup besteht from multiple Schichten:

19.2. Velero Backup Faulty

velero backup get
velero backup describe <backup> --details
velero backup logs <backup>
velero restore get
velero restore describe <restore> --details
kubectl -n velero logs deploy/velero --tail=300

Check:

• Object-Storage-Credentials,
• Bucket/Container reachable,
• Volume-Snapshot-Plugin installed,
• CRDs present,
• Namespace-Filter,
• resource-Excludes,
• Restic/Kopia Node Agent,
• Cloud-API-Quotas,
• IAM/RBAC.

19.3. etcd Restore Fails

Check before the Restore:

• snapshot integrity,
• Kubernetes-Version,
• etcd-Version,
• Cluster-ID and Member-ID,
• certificates,
• dataverzeichnis,
• Static-Pod-Manifeste,
• Quorum-plarrives.

ETCDCTL_API=3 etcdctl snapshot status snapshot.db --write-out=table

Restore is a kontrollierte restore, no Live-Fix. In HA-Clustern need you a konsistenten plan for all Control-Plane-Nodes.

19.4. Restore Successful, but App Does Not Work

Check:

• CRDs before CRs restored?
• Secrets in the correct Namespace?
• ServiceAccount-Tokens new generiert?
• PVs korrekt bound?
• DNS- and Ingress-names zeigen on new Cluster?
• ExternalDNS has Records updated?
• Do certificates and Issuer work?
• databankmigrationen kompatibel?
• App expected old Cluster-IP or Node-IP?

20. Upgrade, Patch, and Maintenance Issues

20.1. Upgrade Precheck

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get pdb -A
kubectl get apiservice
kubectl get crd
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl api-resources --verbs=list -o name | xargs -n 1 kubectl get --ignore-not-foand -A >/dev/null

Also check:

• Version-Skew,
• deprecated APIs,
• CNI/CSI/Ingress compatibility,
• Add-on-versions,
• Webhooks,
• PDBs,
• node capacity for drain,
• Backup-status,
• Rollback- or Ersatzcluster-Strategie.

20.2. Drain Hangs

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=10m
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
kubectl get pdb -A
kubectl describe pdb <pdb> -n <namespace>

Causes:

• PodDisruptionBudget erlaubt no Eviction,
• Singleton-Pod without Replikat,
• localr emptyDir-State,
• DaemonSet ignorieren vergessen,
• StatefulSet can not umziehen,
• finalizer blocks deletion,
• toleration keeps the Pod on the node.

20.3. CNI Upgrade Breaks Networking

Vorher:

kubectl get nodes -o wide
kubectl get pods -A -o wide | grep -i -E 'cilium|calico|flannel|antrea|canal'
kubectl get networkpolicy -A

During/afterwards:

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -200
kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash

Validiere:

• Pod-zu-Service,
• Pod-to-Pod across nodes,
• DNS,
• NetworkPolicy-Enforcement,
• NodePort/LoadBalancer,
• Ingress/Gateway,
• Service Mesh.

20.4. API Deprecations

kubectl api-resources
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis || true

Also use manifest scanning in CI:

kubectl apply --dry-run=server -f manifests/
pluto detect-files -d manifests/ 2>/dev/null || true
kubent 2>/dev/null || true

21. Environment-Specific Troubleshooting

21.1. WSL2 on Windows 11

WSL-issues liegen oft not in Kubernetes itself, sondern in the combination from Windows-Host, WSL-Distribution, Docker Desktop, NAT/Mirrored Networking, Firewall, DNS and hosts file.

21.1.1. Basic Check

PowerShell:

wsl -l -v
wsl --status
wsl --version
docker version
docker context ls
Test-NetConnection localhost -Port 80
Get-Content C:\Windows\System32\drivers\etc\hosts

WSL:

uname -a
cat /etc/os-release
cat /etc/wsl.conf
cat /mnt/c/Users/$USER/.wslconfig 2>/dev/null || true
ip addr
ip route
cat /etc/resolv.conf
ss -tulpn
docker version
docker context ls
kubectl config get-contexts

21.1.2. Edge Cannot Reach Local Endpoints

If Edge runs on Windows, the hostname and port must be reachable from Windows. A hostname only in /etc/hosts inside WSL does not help Edge.

Check Windows hosts file from WSL:

sudo powershell.exe -Command "Start-Process notepad C:\\Windows\\System32\\drivers\\etc\\hosts -Verb runAs"
cat /mnt/c/Windows/System32/drivers/etc/hosts

Typical entry for local Ingress tests:

127.0.0.1 demo.local api.demo.local grafana.demo.local

If your cluster is reachable through a Minikube IP or VM IP, enter that IP:

192.168.56.20 demo.local api.demo.local

Check:

Resolve-DnsName demo.local
Test-NetConnection demo.local -Port 80
curl.exe -v http://demo.local/

21.1.3. WSL Port Is Not Reachable from Windows

Check, whether the Prozess in WSL on 0.0.0.0 or 127.0.0.1 listens:

ss -tulpn | grep -E ':80|:443|:8080|:3000'

Check Windows Firewall and Portproxy only, if the automatische localhost-Weiterleitung not applies:

netsh interface portproxy show all
Get-NetFirewallRule | ? DisplayName -Match "WSL|Docker|Kubernetes"

WSL new start:

wsl --shutdown

21.1.4. Docker Desktop WSL Integration Defective

docker ps
docker info
ls -l /var/run/docker.sock

Check in Docker Desktop:

• WSL Integration aktiv,
• correct Distribution aktiviert,
• resourceslimit ausreichend,
• Kubernetes-Cluster not parallel in the wrong Context.

21.2. kind

21.2.1. Cluster Does Not Start

kind version
kind get clusters
docker ps -a
docker logs <kind-node-container>
kind export logs ./kind-logs

Check:

• Docker runs,
• enough CPU/RAM/Disk,
• Port 6443 frei,
• Proxy/Registry reachable,
• Node-Image-Version matches,
• Docker network not kollidiert.

21.2.2. Ingress or Service Is Not Reachable

kind ports must be set during creation:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP

You cannot cleanly add Docker port mappings to the same node container afterwards. Recreate the cluster if the port is missing.

kind delete cluster --name workshop
kind create cluster --name workshop --config kind.yaml

21.3. minikube

21.3.1. Startup Problems

minikube version
minikube status
minikube start --driver=docker
minikube logs --file=minikube.log
minikube profile list

Check:

• passender Driver,
• Virtualisierung aktiv,
• Docker/Podman/VirtualBox/VMware reachable,
• enough CPU/RAM,
• Proxy and Registry,
• alter Profilzustand.

21.3.2. LoadBalancer or Ingress Not Reachable

minikube tunnel
minikube service list
minikube addons list
minikube addons enable ingress
minikube ip
kubectl get ingress -A

For local hostnames:

<minikube-ip> demo.local api.demo.local

On Windows, this entry belongs in C:\Windows\System32\drivers\etc\hosts if Edge is used.

21.4. k3d

k3d cluster list
k3d kubeconfig get <cluster>
docker ps --format 'table {{.Names}}\t{{.Ports}}\t{{.status}}'
docker logs k3d-<cluster>-serverlb

Publish ports through the load balancer:

k3d cluster create workshop \
  --servers 1 \
  --agents 2 \
  --port "80:80@loadbalancer" \
  --port "443:443@loadbalancer"

Typische error:

• Port-Mapping missing,
• Traefik automatisch installed, but wrong expected,
• Registry not verbunden,
• Docker network kollidiert,
• kubeconfig shows on old Cluster.

21.5. Docker Desktop Kubernetes

docker version
docker info
kubectl config get-contexts
kubectl config use-context docker-desktop
kubectl get nodes

Typische Actions:

• reset Kubernetes in Docker Desktop,
• increase resource limit,
• delete old LoadBalancer/Ingress tests,
• Context correct,
• Docker Desktop new start.

21.6. Local VM, Vagrant, and Remote VM

ip addr
ip route
ss -tulpn
sudo ufw status verbose || true
sudo firewall-cmd --list-all || true
sudo systemctl status kubelet containerd
sudo journalctl -u kubelet -n 300 --no-pager

Vagrant:

vagrant status
vagrant ssh <node>
vagrant provision <node>
vagrant reload
vagrant destroy -f

Check:

• NAT/Bridged/Host-only-network,
• IP-Adressen the Nodes,
• Host-Firewall,
• VM-Firewall,
• DNS,
• Snapshot-Version,
• Provider-network,
• MTU,
• Ports 6443, 2379-2380, 10250, 30000-32767.

21.7. Bare Metal

ip link
ethtool <nic>
dmesg -T | grep -i -E 'error|firmware|link|nvme|disk|xfs|ext4|oom'
lsblk
smartctl -a /dev/<disk> 2>/dev/null || true
journalctl -u kubelet -n 300 --no-pager

Check:

• BIOS/UEFI and Virtualisierung,
• VLAN and Switch-Port,
• MTU,
• DHCP or statische IPs,
• DNS and NTP,
• RAID/Controller,
• Disk-Latenz for etcd,
• BMC/IPMI/iDRAC/iLO,
• Kernel-Module for Storage or CNI.

21.8. k3s, RKE2 and k0s

k3s:

sudo systemctl status k3s
sudo journalctl -u k3s -n 300 --no-pager
sudo k3s kubectl get nodes
sudo k3s kubectl -n kube-system get pods

RKE2:

sudo systemctl status rke2-server rke2-agent
sudo journalctl -u rke2-server -n 300 --no-pager
sudo /var/lib/rancher/rke2/bin/kubectl get nodes --kubeconfig /etc/rancher/rke2/rke2.yaml

k0s:

sudo k0s status
sudo k0s kubectl get nodes
sudo journalctl -u k0scontroller -n 300 --no-pager
sudo journalctl -u k0sworker -n 300 --no-pager

Check Token, Server-URL, certificates, Firewall, Embedded-etcd-state, Add-on-Defaults, CNI and Ingress-Controller.

21.9. Talos

Talos has no normale SSH-erroranalyse. You nutzt talosctl.

talosctl version
talosctl health --nodes <node-ip> --endpoints <endpoint-ip>
talosctl service kubelet --nodes <node-ip>
talosctl logs kubelet --nodes <node-ip>
talosctl logs containerd --nodes <node-ip>
talosctl dmesg --nodes <node-ip>
talosctl get members --nodes <control-plane-ip>
talosctl etcd status --nodes <control-plane-ip>
talosctl kubeconfig --nodes <control-plane-ip>

Check:

• Endpoint and Node-Adresse,
• Machine Config,
• Bootstrap-state,
• certificates,
• etcd-Member,
• network and DNS,
• Disk-Auswahl,
• Control-Plane- and Worker-roles.

22. Cloud-Specific Troubleshooting

22.1. AWS EKS

Basic Check

aws sts get-caller-identity
aws eks describe-cluster --name <cluster> --region <region>
aws eks update-kubeconfig --name <cluster> --region <region>
kubectl get nodes -o wide
kubectl get pods -A -o wide
aws eks list-addons --cluster-name <cluster> --region <region>

Common EKS Errors

kubectl -n kube-system get ds aws-node
kubectl -n kube-system logs ds/aws-node --tail=300
kubectl -n kube-system get deploy coredns
kubectl get nodes -o custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INSTANCE:.metadata.labels.node\.kubernetes\.io/instance-type

22.2. Azure AKS

az account show
az aks show -g <rg> -n <cluster>
az aks get-credentials -g <rg> -n <cluster> --overwrite-existing
az aks nodepool list -g <rg> --cluster-name <cluster>
kubectl get nodes -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -100

Common AKS-error:

• Managed Identity has no permissions,
• Node Pool Upgrade hangs an PDBs,
• Azure CNI Subnet has no free IPs,
• Standard Load Balancer or NSG blocks,
• Disk CSI or File CSI missing,
• private cluster is accessed from outside the network,
• quota for VM family exhausted.

az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool>
az vmss list -g <node-resource-group> -o table
az network nsg list -g <node-resource-group> -o table

22.3. Google GKE

gcloud auth list
gcloud config list
gcloud container clusters describe <cluster> --region <region>
gcloud container clusters get-credentials <cluster> --region <region>
gcloud container operations list --region <region>
kubectl get nodes -o wide

Common GKE-error:

• IAM-Role missing,
• VPC-native IP ranges exhausted,
• Autopilot lehnt Pod specification ab,
• Workload Identity wrong bound,
• Cloud Logging API deaktiviert,
• firewall rules block traffic,
• Ingress/Gateway Controller has no permission or wrong Annotation,
• Persistent Disk CSI Zone matches not.

gcloud compute firewall-rules list
gcloud compute networks subnets describe <subnet> --region <region>
gcloud container node-pools list --cluster <cluster> --region <region>

22.4. DigitalOcean DOKS, Linode LKE, Oracle OKE, IBM IKS

These providers differ in CLI and managed add-ons, but the error classes are similar:

• kubeconfig and API-access,
• Node-Pool-state,
• cloud firewall,
• LoadBalancer-Controller,
• CSI-Driver,
• Quotas,
• Registry-access,
• private Cluster-networks,
• version compatibility.

General pattern:

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get svc -A -o wide
kubectl get pvc -A
kubectl get events -A --sort-by=.lastTimestamp | tail -200

22.5. OpenShift/OKD

oc whoami
oc status
oc get clusterversion
oc get co
oc get nodes
oc get pods -A -o wide
oc adm must-gather

Common error:

• ClusterOperator Degraded,
• MachineConfigPool hangs,
• SCC blocks Pods,
• Route or Router-issues,
• ImageStream or Registry-issues,
• Operator Lifecycle Manager blocks Installation,
• etcd Operator reports quorum or disk issues.

23. Runbook Catalog by Symptom

The following runbooks are deliberately structured for reuse: symptom, causes, diagnosis, actions, and validation. Use them as a checklist during an incident or as exercises in the workshop.

23.1. API Server Not Reachable

Symptom
kubectl returns timeout, connection refused, or TLS error.

Common Causes

• wrong Context.
• VPN or Firewall blocks.
• API Server Pod down.
• Load-Balancer-Endpoint wrong.
• certificate expired.

Diagnosis

kubectl cluster-info
kubectl get --raw='/readyz?verbose'
nc -vz <api-server> 6443
kubectl config view --minify --raw

Actions

• Correct the context.
• Check the network path.
• Check control-plane logs.
• Check certificates.
• Check provider status.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.2. kubectl Reports Forbidden

Symptom
API reachable, but action is denied.

Common Causes

• wrong user.
• missing RoleBinding.
• wrong Namespace.
• Subresource missing.
• Cloud-IAM and RBAC verwechselt.

Diagnosis

kubectl auth whoami
kubectl auth can-i <verb> <resource> -n <namespace>
kubectl get role,rolebinding -n <namespace> -o yaml

Actions

• Add a RoleBinding.
• Allow the subresource.
• Correct the ServiceAccount.
• Avoid ClusterRoleBinding when a namespace-scoped Role is sufficient.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.3. Pod Pending

Symptom
Pod was created, but not on a Node gestarts.

Common Causes

• too little CPU or Memory.
• PVC unbound.
• taint without toleration.
• wrong NodeSelector.
• affinity not satisfiable.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl get nodes -o wide
kubectl get pvc -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Actions

• Adjust requests.
• Expand capacity.
• Correct the PVC/StorageClass.
• Check toleration or NodeSelector.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.4. Pod ContainerCreating Hangs

Symptom
Pod is scheduled, but the container is not fully created.

Common Causes

• Image-Pull langsam.
• Volume-Mount faulty.
• CNI Sandbox faulty.
• Runtime hangs.
• Secret or ConfigMap missing.

Diagnosis

kubectl describe pod <pod> -n <namespace>
journalctl -u kubelet -n 300 --no-pager
crictl ps -a
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Actions

• Resolve mount errors.
• Check CNI.
• Recheck the runtime.
• Correct image/Secret.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.5. ImagePullBackOff

Symptom
Kubernetes can the Container Image not ziehen.

Common Causes

• Image-Name wrong.
• Tag missing.
• Registry not reachable.
• ImagePullSecret wrong.
• Rate Limit.
• wrong Architektur.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
crictl pull <image>
docker manifest inspect <image>

Actions

• Correct the image.
• Recreate the Secret.
• Check registry networking.
• Pin the digest.
• Check architecture.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.6. CrashLoopBackOff

Symptom
Container starts, terminates, and restarts repeatedly.

Common Causes

• App-error.
• Config missing.
• Secret missing.
• wrong Entrypoint.
• Dependency not reachable.
• Liveness probe kills the app.

Diagnosis

kubectl logs <pod> -n <namespace> --previous
kubectl describe pod <pod> -n <namespace>
kubectl get deploy <deploy> -n <namespace> -o yaml

Actions

• Evaluate the exit code.
• Check Config/Secret.
• Relax the probe.
• Check the dependency.
• Perform a rollback.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.7. OOMKilled

Symptom
Container was wegen memoryverbrauch terminates.

Common Causes

• Memory-Limit zu niedrig.
• Memory-Leak.
• too large Cache.
• Sidecar-Verbrauch.
• Node MemoryPressure.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl top pod <pod> -n <namespace>
kubectl top nodes
kubectl logs <pod> -n <namespace> --previous

Actions

• Check the memory profile.
• Adjust limit/request.
• Configure cache.
• Fix leak.
• Expand node capacity.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.8. Readiness Remains false

Symptom
Pod runs, bekommt but no Service-Traffic.

Common Causes

• Probe-Pfad wrong.
• Port wrong.
• App takes longer.
• Dependency missing.
• TLS/Auth in Probe wrong.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl exec -it <pod> -n <namespace> -- wget -S -O- http://127.0.0.1:<port>/<path>
kubectl get endpoints <svc> -n <namespace>

Actions

• Correct the probe.
• Adjust initialDelay.
• Provide the dependency.
• Check Service targetPort.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.9. Liveness Probe Causes Restarts

Symptom
Pod restarts repeatedly even though the app could basically run.

Common Causes

• Probe zu aggressiv.
• Startphase not abgedeckt.
• wrong Pfad.
• Timeout zu kurz.
• CPU throttling.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get pod <pod> -n <namespace> -o yaml | yq '.spec.containers[].livenessProbe'

Actions

• Add a startup probe.
• Increase timeout/period.
• Correct the path.
• Adjust resources.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.10. InitContainer Hangs

Symptom
InitContainer does not complete or fails repeatedly.

Common Causes

• Migration hangs.
• Dependency missing.
• Secret missing.
• network blocks.
• ImagePull-error.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <init-container>
kubectl get pod <pod> -n <namespace> -o yaml

Actions

• Evaluate init logs.
• Test the dependency.
• Define a timeout.
• Ensure idempotency.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.11. Deployment Rollout Hangs

Symptom
New ReplicaSets do not become fully ready.

Common Causes

• new Pods not ready.
• PDB/Quota.
• maxUnavailable/maxSurge.
• Image or Config defective.
• Admission blocks.

Diagnosis

kubectl rollout status deploy/<deploy> -n <namespace>
kubectl rollout history deploy/<deploy> -n <namespace>
kubectl describe deploy <deploy> -n <namespace>
kubectl get rs,pod -n <namespace> -l app=<app>

Actions

• Check faulty ReplicaSet Pods.
• Rollback.
• Adjust the strategy.
• Check quota/capacity.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.12. StatefulSet Hangs

Symptom
One ordinal Pod blocks subsequent Pods.

Common Causes

• PVC/Mount-error.
• Readiness of an earlier ordinal.
• DNS Headless Service.
• datamigration.
• PodManagementPolicy.

Diagnosis

kubectl get sts,pod,pvc -n <namespace>
kubectl describe sts <sts> -n <namespace>
kubectl describe pod <sts>-0 -n <namespace>

Actions

• Resolve the first faulty ordinal Pod.
• Check PVC.
• Readiness correct.
• Check data consistency.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.13. DaemonSet Does Not Run on All Nodes

Symptom
DaemonSet-Pods missing on bestimmten Nodes.

Common Causes

• Taints.
• NodeSelector.
• Affinity.
• HostPort-Konflikt.
• OS/Architektur matches not.

Diagnosis

kubectl get ds -A
kubectl describe ds <ds> -n <namespace>
kubectl get nodes --show-labels

Actions

• Add tolerations.
• Correct the selector.
• Check HostPort.
• Use a multi-architecture image.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.14. Job Fails

Symptom
Job reaches no successful completion.

Common Causes

• Exit Code.
• BackoffLimit reaches.
• Deadline exceeded.
• Secret missing.
• datamigration not idempotent.

Diagnosis

kubectl get job -n <namespace>
kubectl describe job <job> -n <namespace>
kubectl logs job/<job> -n <namespace> --all-containers

Actions

• Evaluate error logs.
• Adjust backoff/deadline.
• Make the Job idempotent.
• Check data state.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.15. CronJob Does Not Run

Symptom
Scheduled Job is not created or runs too late.

Common Causes

• Schedule wrong.
• Suspend=true.
• ConcurrencyPolicy.
• StartDeadlineSeconds.
• time zone.
• Controller disrupted.

Diagnosis

kubectl get cronjob -n <namespace>
kubectl describe cronjob <cronjob> -n <namespace>
kubectl get jobs -n <namespace>

Actions

• Correct the schedule.
• Disable suspend.
• Check timezone.
• Check concurrencyPolicy.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.16. Service has no Endpoints

Symptom
Service exists but routes to no Pod.

Common Causes

• Selector does not match.
• Pods not ready.
• wrong Namespace.
• EndpointSlice-Controller.
• Readiness Gate.

Diagnosis

kubectl get svc,endpoints,endpointslices -n <namespace>
kubectl get pods -n <namespace> --show-labels
kubectl describe pod <pod> -n <namespace>

Actions

• Correct labels/selector.
• Resolve readiness.
• Check the Service port.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.17. ClusterIP timeout

Symptom
Service has Endpoints, but ClusterIP does not respond.

Common Causes

• NetworkPolicy.
• kube-proxy/eBPF.
• CNI.
• wrong targetPort.
• App does not listen.

Diagnosis

kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash
kubectl get endpointslices -n <namespace>
kubectl get networkpolicy -A

Actions

• Adjust policy.
• Check CNI/kube-proxy.
• Correct targetPort.
• Check app port.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.18. DNS Defective in the Cluster

Symptom
Service names cannot be resolved.

Common Causes

• CoreDNS down.
• NetworkPolicy blocks DNS.
• CoreDNS ConfigMap wrong.
• NodeLocal DNS Cache.
• resolv.conf defective.

Diagnosis

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs deploy/coredns --tail=200
kubectl run dnsutils --rm -it --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- nslookup kubernetes.default

Actions

• Scale CoreDNS.
• Allow DNS egress.
• Correct ConfigMap.
• Check CNI.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.19. Ingress 404

Symptom
Ingress-Controller responds, but Route matches not.

Common Causes

• Host wrong.
• Path wrong.
• IngressClass wrong.
• Default Backend.
• Annotation wrong.

Diagnosis

kubectl describe ingress <ingress> -n <namespace>
curl -H 'Host: <host>' http://<ip>/<path>
kubectl logs -n <ingress-namespace> deploy/<controller> --tail=300

Actions

• Correct host/path.
• Set ingressClass.
• Check DNS/hosts file.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.20. Ingress 503

Symptom
Ingress findet no bereiten Backends.

Common Causes

• Service without Endpoints.
• Pods not ready.
• targetPort wrong.
• Backend-Protokoll wrong.
• NetworkPolicy.

Diagnosis

kubectl get svc,endpoints,endpointslices -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl describe pod <pod> -n <namespace>

Actions

• Restore endpoints.
• Resolve readiness.
• Correct port/protocol.
• Policy allow.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.21. Gateway API Route Not Accepted

Symptom
HTTPRoute or Gateway is not programmed.

Common Causes

• GatewayClass without Controller.
• ParentRef wrong.
• Listener-Konflikt.
• ReferenceGrant missing.
• Service-Referenz wrong.

Diagnosis

kubectl get gatewayclass,gateway,httproute -A
kubectl describe gateway <gateway> -n <namespace>
kubectl describe httproute <route> -n <namespace>

Actions

• Read conditions.
• Install controller.
• Correct parentRefs.
• Add ReferenceGrant.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.22. TLS Certificate Wrong

Symptom
Browser or client reports certificate errors.

Common Causes

• wrong Secret.
• CN/SAN matches not.
• Issuer defective.
• old certificateskette.
• SNI wrong.

Diagnosis

kubectl get secret <tls-secret> -n <namespace> -o yaml
openssl s_client -connect <host>:443 -servername <host> -showcerts
kubectl get certificate,order,challenge -A

Actions

• Reissue certificate.
• Reference Secret.
• Correct DNS/SNI.
• Check trust store.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.23. PVC Pending

Symptom
Claim is not bound.

Common Causes

• StorageClass missing.
• CSI missing.
• Quota.
• Topology.
• AccessMode.

Diagnosis

kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass -o yaml
kubectl get pods -A | grep -i csi

Actions

• Set StorageClass.
• Install CSI.
• Increase quota.
• Check topology/zone.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide

Environment Notes

• In local container clusters such as kind or k3d, also check Docker containers, port mappings, and Docker networks.
• In WSL, also check Windows Firewall, the Windows hosts file, and Docker Desktop integration.
• In remote VMs and bare metal, also check the OS firewall, routing, VLAN, MTU, DNS, and NTP.
• In managed Kubernetes, also check provider events, IAM, quotas, subnets, load balancers, and managed add-ons.

23.24. Volume Mount Fails

Symptom
Pod hangs wegen Volume-error.

Common Causes

• Secret/ConfigMap missing.
• PV not attachbar.
• Permission denied.
• Multi-Attach.
• Node-Plugin defective.

Diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl get volumeattachment
kubectl logs -n <csi-ns> ds/<csi-node> --tail=200

Actions

• Create the missing resource.
• Resolve attachment.
• Correct permissions.
• Check CSI.

Validation

kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get nodes -o wide