Command Reference: Kubernetes
Last updated: June 13, 2026
Target audience: Kubernetes workshops, operations, troubleshooting, reviews, and final projects
This reference collects the most important Kubernetes commands and explains when to use them, what output to expect, and what to pay attention to. The focus is on kubectl, kubeadm, and the tools you regularly need around Kubernetes in real operations: kubelet, control-plane binaries, crictl, ctr, nerdctl, etcdctl, etcdutl, Helm, Kustomize, CNI tools, cluster tools, and Linux helper commands.
Important: Kubernetes commands are highly dependent on version, distribution, cloud provider, and installed add-ons. This file is therefore designed as an extensive workshop and operations reference. For the complete flag reference of a specific installation, always also check
<command> --help,<command> <subcommand> --help, and the documentation for your Kubernetes version.
1. Basic Rules for Safe Work
Kubernetes commands often act directly on production workloads. Work in a clear sequence:
- Check the context: First verify cluster, namespace, and identity.
- Read before writing: Start with
get,describe,diff,dry-run, andauth can-i. - Limit the change: Work with a namespace, label selector, or specific object instead of
-Aor--all. - Understand the impact: Evaluate whether a command restarts Pods, redirects traffic, deletes data, or drains Nodes.
- Prepare rollback: Save the manifest, values, revision, backup, or snapshot beforehand.
- Validate afterwards: Check status, Events, logs, Endpoints, and metrics.
Dangerous Command Classes
| Command class | Risk | Safer handling |
|---|---|---|
kubectl delete |
Removes resources or triggers re-creation. | Run kubectl get first, then delete precisely. Avoid uncontrolled wildcards. |
kubectl replace --force |
Deletes and re-creates the object. This can cause downtime. | Use only when you intentionally want the re-creation effect. |
kubectl drain |
Evicts Pods from a Node. | Check PDBs, DaemonSets, local data, and capacity first. |
kubeadm reset |
Removes local cluster artifacts from a Node. | Run only on the correct host. Check CNI, iptables, and data leftovers afterwards. |
etcdctl snapshot restore / etcdutl snapshot restore |
Creates a new logical etcd cluster state. | Use only with a documented restore plan and backup. |
kubectl patch |
Can overwrite fields without you noticing. | Keep patches small and verify the result with get -o yaml. |
kubectl apply --prune |
Can remove resources that are no longer declared. | Use only with a narrow selector and a test environment. |
2. Notation and Placeholders
This file uses placeholders consistently:
| Placeholder | Meaning | Example |
|---|---|---|
<namespace> |
Namespace | demo |
<pod> |
Pod name | web-785d9c9d8f-abc12 |
<deployment> |
Deployment name | web |
<service> |
Service name | web |
<node> |
Node name | worker-01 |
<container> |
Container name in a Pod | app |
<context> |
kubeconfig context | dev-cluster |
<image> |
Container image | ghcr.io/example/app:1.2.3 |
<file> |
Manifest, values file, or snapshot | deployment.yaml |
Conventions:
# Set namespace explicitly
kubectl get pods -n <namespace>
# Show all namespaces
kubectl get pods -A
kubectl get pods --all-namespaces
# Short resource names
kubectl get po,svc,deploy -n <namespace>
# Long resource names are often clearer in documentation and workshops
kubectl get pods,services,deployments -n <namespace>
3. Shell, Autocomplete, and Aliases
Check Versions
kubectl version --client
kubectl version
kubectl version -o yaml
kubeadm version
helm version
crictl --version
ctr version
etcdctl version
kubectl version --client checks only your local CLI. kubectl version also contacts the API Server. If you do not have access, you will see authentication, network, or certificate errors here.
Autocomplete for Bash
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
alias k=kubectl
complete -o default -F __start_kubectl k
Autocomplete for Zsh
source <(kubectl completion zsh)
echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc
Autocomplete for Fish
mkdir -p ~/.config/fish/completions
kubectl completion fish > ~/.config/fish/completions/kubectl.fish
Useful Aliases
alias k='kubectl'
alias kgp='kubectl get pods'
alias kgn='kubectl get nodes -o wide'
alias kga='kubectl get all'
alias kaf='kubectl apply -f'
alias kdf='kubectl diff -f'
alias kdel='kubectl delete'
alias kctx='kubectl config current-context'
alias kns='kubectl config set-context --current --namespace'
Use aliases only interactively. In documentation, runbooks, and CI pipelines, fully written commands are more robust.
4. kubectl: Syntax and Global Options
kubectl is the standard CLI for communicating with the Kubernetes Control Plane. The basic form is:
kubectl [global options] <verb> <resource> [name] [flags]
Examples:
kubectl get pods -n demo
kubectl describe deployment web -n demo
kubectl apply -f deployment.yaml
kubectl delete service web -n demo
Global Options
| Option | Purpose | Example |
|---|---|---|
--kubeconfig <file> |
Uses a specific kubeconfig file. | kubectl --kubeconfig ./kubeconfig get nodes |
--context <context> |
Uses a specific context. | kubectl --context prod get pods -A |
-n, --namespace |
Sets the namespace for namespaced resources. | kubectl get pods -n demo |
-A, --all-namespaces |
Shows resources across all namespaces. | kubectl get pods -A |
-o <format> |
Output format. | kubectl get pods -o wide |
-l, --selector |
Filters by label selector. | kubectl get pods -l app=web |
--field-selector |
Filters by field. | kubectl get pods --field-selector=status.phase=Pending |
--as <user> |
Impersonates a user. | kubectl auth can-i get pods --as alice |
--as-group <group> |
Impersonates a group. | kubectl auth can-i list pods --as-group devs |
--request-timeout <duration> |
Timeout for API requests. | kubectl get pods --request-timeout=10s |
--v <level> |
Increases verbosity for debugging. | kubectl get pods --v=8 |
--dry-run=client |
Validates or generates locally without API persistence. | kubectl create ns demo --dry-run=client -o yaml |
--dry-run=server |
Validates through the API Server, including Admission. | kubectl apply --dry-run=server -f app.yaml |
Output Formats
kubectl get pods -o wide
kubectl get pods -o yaml
kubectl get pods -o json
kubectl get pods -o name
kubectl get pods -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase'
kubectl get pods -o jsonpath='{.items[*].metadata.name}'
Common Short Resource Names
| Short name | Long name |
|---|---|
po |
pods |
svc |
services |
deploy |
deployments |
rs |
replicasets |
sts |
statefulsets |
ds |
daemonsets |
cm |
configmaps |
sa |
serviceaccounts |
ns |
namespaces |
ep |
endpoints |
pvc |
persistentvolumeclaims |
pv |
persistentvolumes |
sc |
storageclasses |
ing |
ingresses |
netpol |
networkpolicies |
cj |
cronjobs |
5. kubeconfig, Context, and Access
Check the Current Context
kubectl config current-context
kubectl config get-contexts
kubectl cluster-info
kubectl get nodes -o wide
Run these commands before every change. A wrong context is one of the most common operational mistakes.
Switch Context
kubectl config use-context <context>
kubectl config current-context
Set the Namespace in the Current Context
kubectl config set-context --current --namespace=<namespace>
kubectl config view --minify | grep namespace
This is convenient, but it can be dangerous. In runbooks, -n <namespace> is more explicit.
View and Check kubeconfig
kubectl config view
kubectl config view --raw
kubectl config view --minify
kubectl config get-clusters
kubectl config get-users
kubectl config get-contexts
--raw can print sensitive data such as certificates or tokens. Do not share this output without reviewing it first.
Set Cluster, User, and Context
kubectl config set-cluster <cluster> --server=https://<api-server>:6443 --certificate-authority=<ca.crt>
kubectl config set-credentials <user> --client-certificate=<user.crt> --client-key=<user.key>
kubectl config set-context <context> --cluster=<cluster> --user=<user> --namespace=<namespace>
kubectl config use-context <context>
Rename or Delete Contexts
kubectl config rename-context <old-context> <new-context>
kubectl config delete-context <context>
kubectl config delete-cluster <cluster>
kubectl config delete-user <user>
Use a Temporary kubeconfig
KUBECONFIG=./kubeconfig kubectl get nodes
kubectl --kubeconfig ./kubeconfig get pods -A
Merge Multiple kubeconfig Files
export KUBECONFIG=$HOME/.kube/config:./cluster-a.yaml:./cluster-b.yaml
kubectl config view --merge --flatten > ./merged-kubeconfig.yaml
6. API Exploration and Resource Model
Show API Versions and Resources
kubectl api-versions
kubectl api-resources
kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false
kubectl api-resources --verbs=list
kubectl api-resources --api-group=apps
kubectl api-resources -o wide
These commands show which resource types are available in the cluster. CRDs installed by add-ons also appear here.
Explain Resource Schemas
kubectl explain pod
kubectl explain pod.spec
kubectl explain pod.spec.containers
kubectl explain deployment.spec.strategy
kubectl explain networkpolicy.spec.ingress
kubectl explain gateway.spec.listeners
kubectl explain --recursive deployment.spec.template.spec
kubectl explain is the fastest way to understand fields directly from the API schema.
Raw API Access
kubectl get --raw /api
kubectl get --raw /apis
kubectl get --raw /version
kubectl get --raw /healthz
kubectl get --raw /readyz?verbose
kubectl get --raw /livez?verbose
kubectl get --raw /metrics | head
/metrics, /readyz, and /livez may be protected depending on RBAC and API Server configuration.
Subresources
Some resources have subresources such as status, scale, or, depending on the version, resize.
kubectl get deployment web -n demo --subresource=scale -o yaml
kubectl patch deployment web -n demo --subresource=scale --type=merge -p '{"spec":{"replicas":3}}'
kubectl get pod <pod> -n demo --subresource=status -o yaml
Use subresources deliberately when you only want to inspect or modify status, scaling, or another specialized part of an object.
7. Viewing, Filtering, and Formatting Resources
Basic Commands
kubectl get namespaces
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get all -n <namespace>
kubectl get deployments,replicasets,pods -n <namespace>
kubectl get services,endpoints,endpointslices -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl describe node <node>
Watch Mode
kubectl get pods -n <namespace> -w
kubectl get events -n <namespace> --watch
watch -n 2 'kubectl get pods -n <namespace>'
kubectl -w follows API Events. watch repeatedly runs a command and is useful when you want to display multiple object types together.
Label Selectors
kubectl get pods -n demo -l app=web
kubectl get pods -n demo -l 'app in (web,api)'
kubectl get pods -n demo -l 'app=web,environment=dev'
kubectl get pods -A -l 'app.kubernetes.io/name=nginx'
Field Selectors
kubectl get pods -A --field-selector=status.phase=Pending
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --field-selector=type=Warning
kubectl get pods -A --field-selector=spec.nodeName=<node>
Field selectors are more limited than label selectors, but they are very useful for status- and Node-related queries.
Sorting
kubectl get pods -A --sort-by=.metadata.creationTimestamp
kubectl get events -A --sort-by=.lastTimestamp
kubectl get pv --sort-by=.spec.capacity.storage
kubectl get pods -A --sort-by=.status.containerStatuses[0].restartCount
Output as Names
kubectl get pods -n demo -o name
kubectl get deploy -n demo -o name
kubectl get pods -n demo -l app=web -o name | xargs -r -n1 kubectl describe -n demo
Save YAML and JSON Selectively
kubectl get deployment web -n demo -o yaml > deployment.web.yaml
kubectl get service web -n demo -o json > service.web.json
Inspect Multiple Resources Together
kubectl get deploy,rs,po,svc,ep,endpointslice -n demo -o wide
kubectl get ingress,svc,ep,endpointslice -n demo -o wide
kubectl get pvc,pv,storageclass -A
8. Creating, Applying, and Validating Resources
Declarative Work with apply
kubectl apply -f manifest.yaml
kubectl apply -f ./manifests/
kubectl apply -f https://example.invalid/manifest.yaml
kubectl apply -k overlays/dev
apply is the default for declarative work. The manifest file remains the authoritative desired state.
Server-Side Dry Run
kubectl apply --dry-run=server -f manifest.yaml
kubectl apply --dry-run=server -k overlays/dev
Server-side dry run validates against the API Server and includes Admission Controllers. This is more meaningful than purely local YAML validation.
Client-Side Dry Run and YAML Generation
kubectl create namespace demo --dry-run=client -o yaml
kubectl create deployment web --image=nginx:1.27 --replicas=2 --dry-run=client -o yaml
kubectl create service clusterip web --tcp=80:8080 --dry-run=client -o yaml
kubectl run debug --image=busybox --restart=Never --dry-run=client -o yaml -- sleep 3600
Use this form to generate starter manifests and then version them cleanly.
Diff Before a Change
kubectl diff -f manifest.yaml
kubectl diff -k overlays/dev
kubectl diff -f ./manifests/ --server-side
diff shows which fields would change in the cluster.
Server-Side Apply
kubectl apply --server-side -f manifest.yaml
kubectl apply --server-side --field-manager=workshop -f manifest.yaml
kubectl apply --server-side --force-conflicts -f manifest.yaml
Server-side apply is useful when multiple controllers or teams manage fields. Use --force-conflicts only when you deliberately want to override field ownership.
Replace
kubectl replace -f manifest.yaml
kubectl replace --force -f manifest.yaml
replace replaces the resource based on the supplied object. replace --force deletes and re-creates it; this can cause downtime.
Validation with Schema and Admission
kubectl apply --dry-run=server -f manifest.yaml
kubectl explain deployment.spec.template.spec.containers.resources
kubectl get events -n demo --sort-by=.lastTimestamp
If Admission Controllers such as policy engines are active, a manifest can be syntactically valid and still be rejected.
9. Modifying Resources: edit, patch, label, annotate, and set
Interactive Editing
kubectl edit deployment web -n demo
kubectl edit service web -n demo
kubectl edit configmap app-config -n demo
kubectl edit node <node>
edit is useful for emergencies, but it is not very reproducible. Transfer permanent changes back into your Git repository afterwards.
Patch with Merge Patch
kubectl patch deployment web -n demo --type=merge -p '{"spec":{"replicas":3}}'
kubectl patch service web -n demo --type=merge -p '{"spec":{"type":"ClusterIP"}}'
Patch with JSON Patch
kubectl patch deployment web -n demo --type=json -p='[
{"op":"replace","path":"/spec/replicas","value":3}
]'
JSON Patch is precise, but sensitive to paths.
Patch an Annotation
kubectl patch deployment web -n demo --type=merge -p '{
"metadata": {
"annotations": {
"workshop.example/change-reason": "manual-test"
}
}
}'
Set and Remove Labels
kubectl label namespace demo environment=dev
kubectl label pod <pod> -n demo debug=true
kubectl label deployment web -n demo app.kubernetes.io/version=1.2.3 --overwrite
kubectl label namespace demo environment-
Labels control selectors, policies, metric aggregation, and often GitOps or deployment logic. Change them carefully.
Set and Remove Annotations
kubectl annotate deployment web -n demo kubernetes.io/change-cause='image update to 1.2.3'
kubectl annotate pod <pod> -n demo workshop.example/owner='team-a' --overwrite
kubectl annotate deployment web -n demo kubernetes.io/change-cause-
Change Images
kubectl set image deployment/web app=<image> -n demo
kubectl set image deployment/web app=ghcr.io/example/app:1.2.3 -n demo --record=false
kubectl set image daemonset/node-agent agent=ghcr.io/example/agent:2.0.0 -n kube-system
Change Environment Variables
kubectl set env deployment/web LOG_LEVEL=debug -n demo
kubectl set env deployment/web LOG_LEVEL- -n demo
kubectl set env deployment/web --from=configmap/app-config -n demo
kubectl set env deployment/web --list -n demo
Change Resources
kubectl set resources deployment/web -n demo --requests=cpu=100m,memory=128Mi --limits=cpu=500m,memory=512Mi
kubectl set resources deployment/web -n demo -c app --requests=cpu=200m,memory=256Mi
Set ServiceAccount
kubectl set serviceaccount deployment/web app-sa -n demo
Set Selector
kubectl set selector service/web app=web -n demo
Be especially careful with selectors. A wrong Service selector quickly leads to Services without Endpoints or to traffic being sent to the wrong Pods.
10. Deleting, Waiting For, and Cleaning Up Resources
Delete Precisely
kubectl delete pod <pod> -n demo
kubectl delete deployment web -n demo
kubectl delete service web -n demo
kubectl delete -f manifest.yaml
kubectl delete -k overlays/dev
Delete by Label
kubectl delete pods -n demo -l app=web
kubectl delete jobs -n demo -l cleanup=true
Delete All Resources of a Type
kubectl delete pods --all -n demo
kubectl delete jobs --all -n demo
Use --all only in deliberately scoped namespaces.
Grace Period and Force Delete
kubectl delete pod <pod> -n demo --grace-period=30
kubectl delete pod <pod> -n demo --grace-period=0 --force
--force aggressively removes the API object. This does not automatically mean that the process on the Node was terminated cleanly.
Check and Remove Finalizers
kubectl get <resource> <name> -n demo -o jsonpath='{.metadata.finalizers}'
kubectl patch <resource> <name> -n demo --type=merge -p '{"metadata":{"finalizers":null}}'
Remove finalizers only if you understand which controller is blocking the resource. Otherwise external artifacts may be left behind.
Wait for a State
kubectl wait --for=condition=Available deployment/web -n demo --timeout=120s
kubectl wait --for=condition=Ready pod/<pod> -n demo --timeout=60s
kubectl wait --for=delete pod/<pod> -n demo --timeout=60s
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/<pod> -n demo --timeout=60s
Use Prune Carefully
kubectl apply -f ./manifests/ --prune -l app.kubernetes.io/part-of=demo
--prune can delete resources that are no longer included in the applied set. Use a narrow and unambiguous label selector.
11. Pods
Show Pods
kubectl get pods -n demo
kubectl get pods -n demo -o wide
kubectl get pod <pod> -n demo -o yaml
kubectl describe pod <pod> -n demo
Find Pods by Status
kubectl get pods -A --field-selector=status.phase=Pending
kubectl get pods -A --field-selector=status.phase=Failed
kubectl get pods -A --field-selector=status.phase=Succeeded
kubectl get pods -A --field-selector=spec.nodeName=<node>
Show Containers of a Pod
kubectl get pod <pod> -n demo -o jsonpath='{.spec.containers[*].name}'
kubectl get pod <pod> -n demo -o jsonpath='{.spec.initContainers[*].name}'
kubectl get pod <pod> -n demo -o jsonpath='{.status.containerStatuses[*].restartCount}'
Start a Temporary Pod
kubectl run tmp-shell -n demo --rm -it --image=busybox:1.36 --restart=Never -- sh
kubectl run curl -n demo --rm -it --image=curlimages/curl:8.8.0 --restart=Never -- sh
kubectl run dns-test -n demo --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local
Generate a Pod Manifest
kubectl run app -n demo --image=nginx:1.27 --restart=Never --dry-run=client -o yaml > pod.yaml
Delete a Pod So the Controller Re-Creates It
kubectl delete pod <pod> -n demo
kubectl get replicaset,deployment -n demo
If the Pod is managed by a Deployment, ReplicaSet, StatefulSet, or DaemonSet, the controller usually creates a replacement. If the Pod was created manually, it is gone afterwards.
12. Deployments, ReplicaSets, and Rollouts
Create a Deployment
kubectl create deployment web -n demo --image=nginx:1.27 --replicas=2
kubectl create deployment web -n demo --image=nginx:1.27 --replicas=2 --dry-run=client -o yaml > deployment.yaml
Check a Deployment
kubectl get deployment web -n demo
kubectl describe deployment web -n demo
kubectl get replicaset -n demo -l app=web
kubectl get pods -n demo -l app=web -o wide
Start and Watch a Rollout
kubectl set image deployment/web nginx=nginx:1.28 -n demo
kubectl rollout status deployment/web -n demo --timeout=120s
kubectl rollout history deployment/web -n demo
Check Rollout History and Revisions
kubectl rollout history deployment/web -n demo
kubectl rollout history deployment/web -n demo --revision=2
kubectl describe deployment web -n demo
Rollback
kubectl rollout undo deployment/web -n demo
kubectl rollout undo deployment/web -n demo --to-revision=2
kubectl rollout status deployment/web -n demo
Pause and Resume a Rollout
kubectl rollout pause deployment/web -n demo
kubectl set image deployment/web app=ghcr.io/example/app:1.2.4 -n demo
kubectl set env deployment/web FEATURE_X=true -n demo
kubectl rollout resume deployment/web -n demo
kubectl rollout status deployment/web -n demo
Restart a Deployment
kubectl rollout restart deployment/web -n demo
kubectl rollout status deployment/web -n demo
rollout restart sets an annotation in the Pod template and thereby creates new Pods.
Check ReplicaSets
kubectl get rs -n demo
kubectl describe rs <replicaset> -n demo
kubectl get pods -n demo --show-labels
ReplicaSets are normally not modified directly. Modify the Deployment instead.
13. StatefulSets, DaemonSets, Jobs, and CronJobs
StatefulSets
kubectl get statefulsets -n demo
kubectl describe statefulset db -n demo
kubectl rollout status statefulset/db -n demo
kubectl rollout restart statefulset/db -n demo
kubectl scale statefulset/db -n demo --replicas=3
kubectl get pods -n demo -l app=db -o wide
kubectl get pvc -n demo -l app=db
StatefulSets create stable Pod names and frequently use PVCs. When the StatefulSet is deleted, PVCs may remain depending on the storage policy.
DaemonSets
kubectl get daemonsets -A
kubectl describe daemonset <daemonset> -n <namespace>
kubectl rollout status daemonset/<daemonset> -n <namespace>
kubectl rollout restart daemonset/<daemonset> -n <namespace>
kubectl get pods -n <namespace> -l <label-key>=<label-value> -o wide
DaemonSets are typically used for Node-near agents: CNI, CSI, logging, monitoring, and security.
Jobs
kubectl create job migrate -n demo --image=ghcr.io/example/app:1.2.3 -- ./migrate.sh
kubectl create job migrate -n demo --image=ghcr.io/example/app:1.2.3 --dry-run=client -o yaml > job.yaml
kubectl get jobs -n demo
kubectl describe job migrate -n demo
kubectl logs job/migrate -n demo
kubectl delete job migrate -n demo
Create a Job from a CronJob
kubectl create job manual-backup -n demo --from=cronjob/backup
kubectl logs job/manual-backup -n demo
CronJobs
kubectl create cronjob backup -n demo --image=busybox:1.36 --schedule='*/15 * * * *' -- /bin/sh -c 'date; echo backup'
kubectl get cronjobs -n demo
kubectl describe cronjob backup -n demo
kubectl get jobs -n demo --sort-by=.metadata.creationTimestamp
kubectl delete cronjob backup -n demo
Clean Up Failed Jobs
kubectl get jobs -n demo
kubectl delete job -n demo --field-selector=status.successful=1
kubectl delete job -n demo -l cleanup=true
14. Scaling, Autoscaling, and Availability
Manual Scaling
kubectl scale deployment/web -n demo --replicas=3
kubectl scale statefulset/db -n demo --replicas=3
kubectl get deployment web -n demo
Create a HorizontalPodAutoscaler
kubectl autoscale deployment web -n demo --min=2 --max=10 --cpu-percent=70
kubectl get hpa -n demo
kubectl describe hpa web -n demo
HPA requires metrics, usually through the Metrics Server. Without metrics, the HPA remains ineffective or reports errors.
Check HPA
kubectl get hpa -A
kubectl describe hpa <hpa> -n <namespace>
kubectl top pods -n <namespace>
kubectl top pods -n <namespace> --containers
PodDisruptionBudget
kubectl get poddisruptionbudgets -A
kubectl describe pdb web -n demo
kubectl create poddisruptionbudget web -n demo --selector=app=web --min-available=1 --dry-run=client -o yaml
PDBs protect against voluntary disruptions, such as kubectl drain. They do not prevent Node failures.
Check Deployment Availability
kubectl get deployment web -n demo -o jsonpath='{.status.availableReplicas}'
kubectl get deployment web -n demo -o jsonpath='{.status.conditions}'
kubectl describe deployment web -n demo
15. Logs, Exec, Attach, Copy, and Debug
Show Logs
kubectl logs pod/<pod> -n demo
kubectl logs pod/<pod> -n demo -c <container>
kubectl logs pod/<pod> -n demo --all-containers
kubectl logs deployment/web -n demo
kubectl logs deployment/web -n demo --tail=100
kubectl logs deployment/web -n demo --since=10m
kubectl logs deployment/web -n demo -f
kubectl logs pod/<pod> -n demo -c <container> --previous
--previous is important for CrashLoopBackOff, because the current container often exits immediately again.
Logs by Label
kubectl logs -n demo -l app=web --all-containers=true --tail=100
kubectl logs -n demo -l app=web --prefix=true --since=30m
Exec into a Container
kubectl exec -it pod/<pod> -n demo -- sh
kubectl exec -it pod/<pod> -n demo -c <container> -- bash
kubectl exec pod/<pod> -n demo -- printenv
kubectl exec pod/<pod> -n demo -- cat /etc/resolv.conf
Minimal images often do not include bash, curl, nslookup, or a package manager. Use debug containers in that case.
Attach
kubectl attach pod/<pod> -n demo -c <container>
kubectl attach -it pod/<pod> -n demo -c <container>
attach connects to a running process. For shells, exec is usually more appropriate.
Copy Files
kubectl cp demo/<pod>:/path/in/container ./local-file -c <container>
kubectl cp ./local-file demo/<pod>:/tmp/local-file -c <container>
kubectl cp often requires tar inside the container. If tar is missing, use kubectl exec with streams or a debug image.
Ephemeral Debug Container
kubectl debug -it pod/<pod> -n demo --image=busybox:1.36 --target=<container> -- sh
kubectl debug -it pod/<pod> -n demo --image=nicolaka/netshoot --target=<container> -- bash
Ephemeral Containers modify the Pod for debugging purposes. They are not intended for permanent changes.
Create a Debug Copy of a Pod
kubectl debug pod/<pod> -n demo -it --copy-to=<pod>-debug --container=debug --image=busybox:1.36 -- sh
Debug a Node
kubectl debug node/<node> -it --image=ubuntu:24.04
In the debug shell, the host filesystem is typically available under /host. Use this only with clear intent.
16. Port Forwarding, Proxy, and Local Tests
Port Forward to a Pod
kubectl port-forward pod/<pod> -n demo 8080:80
Port Forward to a Service
kubectl port-forward service/web -n demo 8080:80
curl http://127.0.0.1:8080
Port Forward to a Deployment
kubectl port-forward deployment/web -n demo 8080:8080
API Proxy
kubectl proxy --port=8001
curl http://127.0.0.1:8001/api/
curl http://127.0.0.1:8001/apis/apps/v1/namespaces/demo/deployments
kubectl proxy is useful for local development and API exploration. Do not expose it unprotected on shared systems.
Local Service Tests
kubectl run curl -n demo --rm -it --image=curlimages/curl:8.8.0 --restart=Never -- http://web.demo.svc.cluster.local
kubectl run netshoot -n demo --rm -it --image=nicolaka/netshoot --restart=Never -- bash
17. Services, DNS, Endpoints, and EndpointSlices
Show Services
kubectl get services -n demo
kubectl get svc -n demo -o wide
kubectl describe service web -n demo
kubectl get service web -n demo -o yaml
Create a Service
kubectl expose deployment web -n demo --port=80 --target-port=8080 --type=ClusterIP
kubectl create service clusterip web -n demo --tcp=80:8080 --dry-run=client -o yaml
kubectl create service nodeport web -n demo --tcp=80:8080 --dry-run=client -o yaml
kubectl create service loadbalancer web -n demo --tcp=80:8080 --dry-run=client -o yaml
Check Endpoints
kubectl get endpoints web -n demo -o wide
kubectl get endpointslices -n demo -l kubernetes.io/service-name=web
kubectl describe endpointslice -n demo -l kubernetes.io/service-name=web
If a Service has no Endpoints, the selector and Pod labels usually do not match, or the Pods are not Ready.
Test DNS in the Cluster
kubectl run dns-test -n demo --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local
kubectl run dns-test -n demo --rm -it --image=busybox:1.36 --restart=Never -- nslookup web.demo.svc.cluster.local
kubectl run dns-test -n demo --rm -it --image=busybox:1.36 --restart=Never -- cat /etc/resolv.conf
Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
kubectl get configmap coredns -n kube-system -o yaml
Check Service Traffic
kubectl run curl -n demo --rm -it --image=curlimages/curl:8.8.0 --restart=Never -- curl -sv http://web.demo.svc.cluster.local
kubectl run netshoot -n demo --rm -it --image=nicolaka/netshoot --restart=Never -- curl -sv http://web.demo.svc.cluster.local
18. Ingress and Gateway API
Show Ingresses
kubectl get ingress -A
kubectl describe ingress web -n demo
kubectl get ingress web -n demo -o yaml
Check IngressClass
kubectl get ingressclass
kubectl describe ingressclass <ingressclass>
Check Ingress Controller
kubectl get pods -A | grep -i ingress
kubectl get svc -A | grep -i ingress
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=100
Namespace and labels depend on the specific controller.
Show Gateway API Resources
kubectl get gatewayclasses
kubectl get gateways -A
kubectl get httproutes -A
kubectl get grpcroutes -A
kubectl get tlsroutes -A
kubectl get referencegrants -A
Describe Gateway API Resources
kubectl describe gateway <gateway> -n <namespace>
kubectl describe httproute <route> -n <namespace>
kubectl get httproute <route> -n <namespace> -o yaml
Typical Ingress/Gateway Triage
kubectl get ingress,svc,ep,endpointslice -n demo -o wide
kubectl describe ingress web -n demo
kubectl get events -n demo --sort-by=.lastTimestamp
kubectl get pods -A -o wide | grep -i ingress
kubectl logs -n <ingress-namespace> <ingress-controller-pod> --tail=200
19. NetworkPolicies and Network Tests
Show NetworkPolicies
kubectl get networkpolicy -A
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get networkpolicy <policy> -n <namespace> -o yaml
Network Test Pods
kubectl run netshoot -n demo --rm -it --image=nicolaka/netshoot --restart=Never -- bash
kubectl run busybox -n demo --rm -it --image=busybox:1.36 --restart=Never -- sh
kubectl run curl -n demo --rm -it --image=curlimages/curl:8.8.0 --restart=Never -- sh
Test DNS and HTTP
nslookup web.demo.svc.cluster.local
curl -sv http://web.demo.svc.cluster.local
wget -qO- http://web.demo.svc.cluster.local
nc -vz web.demo.svc.cluster.local 80
Pod-to-Pod Test
kubectl get pods -n demo -o wide
kubectl exec -it <pod> -n demo -- sh
ip addr
ip route
cat /etc/resolv.conf
Basic CNI Check
kubectl get pods -n kube-system -o wide
kubectl get daemonsets -n kube-system
kubectl get nodes -o wide
kubectl describe node <node>
If NetworkPolicies do not take effect, first check whether your CNI actually enforces NetworkPolicies.
20. ConfigMaps, Secrets, and ServiceAccounts
Create a ConfigMap
kubectl create configmap app-config -n demo --from-literal=LOG_LEVEL=info
kubectl create configmap app-config -n demo --from-file=application.yaml
kubectl create configmap app-config -n demo --from-env-file=.env
kubectl create configmap app-config -n demo --from-literal=LOG_LEVEL=info --dry-run=client -o yaml > configmap.yaml
Check a ConfigMap
kubectl get configmap app-config -n demo
kubectl describe configmap app-config -n demo
kubectl get configmap app-config -n demo -o yaml
Create a Secret
kubectl create secret generic app-secret -n demo --from-literal=password='dummy'
kubectl create secret generic app-secret -n demo --from-file=credentials.json
kubectl create secret tls web-tls -n demo --cert=tls.crt --key=tls.key
kubectl create secret docker-registry regcred -n demo \
--docker-server=<registry> \
--docker-username=<user> \
--docker-password=<password> \
--docker-email=<email>
Create a Secret Manifest Without Committing Real Values
kubectl create secret generic app-secret -n demo --from-literal=password='dummy' --dry-run=client -o yaml > secret.example.yaml
Real Secrets do not belong in workshop repositories. Use external Secret systems, Sealed Secrets, External Secrets, or the Secret solution used by your platform.
Read a Secret
kubectl get secret app-secret -n demo -o yaml
kubectl get secret app-secret -n demo -o jsonpath='{.data.password}' | base64 -d; echo
Base64 is not encryption. It is only encoding.
Create a ServiceAccount and Check a Token
kubectl create serviceaccount app-sa -n demo
kubectl get serviceaccount app-sa -n demo -o yaml
kubectl create token app-sa -n demo
Set ImagePullSecrets
kubectl patch serviceaccount default -n demo -p '{"imagePullSecrets":[{"name":"regcred"}]}'
kubectl get serviceaccount default -n demo -o yaml
21. RBAC, Authentication, and Authorization
Check Your Own Permissions
kubectl auth can-i get pods -n demo
kubectl auth can-i list secrets -n demo
kubectl auth can-i create deployments -n demo
kubectl auth can-i '*' '*' -n demo
kubectl auth whoami
Check Permissions for a ServiceAccount
kubectl auth can-i get pods -n demo --as system:serviceaccount:demo:app-sa
kubectl auth can-i list secrets -n demo --as system:serviceaccount:demo:app-sa
kubectl auth can-i '*' '*' --as system:serviceaccount:demo:app-sa
Show Roles and Bindings
kubectl get roles,rolebindings -n demo
kubectl get clusterroles,clusterrolebindings
kubectl describe role <role> -n demo
kubectl describe rolebinding <rolebinding> -n demo
kubectl describe clusterrole <clusterrole>
kubectl describe clusterrolebinding <clusterrolebinding>
Create a Role
kubectl create role pod-reader -n demo --verb=get,list,watch --resource=pods
kubectl create role deployment-editor -n demo --verb=get,list,watch,update,patch --resource=deployments.apps
Create a RoleBinding
kubectl create rolebinding read-pods -n demo --role=pod-reader --serviceaccount=demo:app-sa
kubectl create rolebinding read-pods-user -n demo --role=pod-reader --user=alice@example.com
kubectl create rolebinding read-pods-group -n demo --role=pod-reader --group=developers
ClusterRole and ClusterRoleBinding
kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes
kubectl create clusterrolebinding node-reader-binding --clusterrole=node-reader --user=alice@example.com
Cluster-wide permissions are highly sensitive. Create them only when namespace-scoped roles are not sufficient.
Reconcile RBAC from Manifests
kubectl auth reconcile -f rbac.yaml --dry-run=server
kubectl auth reconcile -f rbac.yaml
Show All ServiceAccounts Used by Pods
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,SA:.spec.serviceAccountName'
22. Pod and Container Security
Check SecurityContext
kubectl get pod <pod> -n demo -o jsonpath='{.spec.securityContext}'
kubectl get pod <pod> -n demo -o jsonpath='{.spec.containers[*].securityContext}'
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,SA:.spec.serviceAccountName'
Find Privileged Pods
kubectl get pods -A -o json | jq -r '
.items[] |
select(any(.spec.containers[]?; .securityContext.privileged == true)) |
[.metadata.namespace, .metadata.name] | @tsv'
Find Pods with Host Network, Host PID, or Host IPC
kubectl get pods -A -o json | jq -r '
.items[] |
select(.spec.hostNetwork == true or .spec.hostPID == true or .spec.hostIPC == true) |
[.metadata.namespace, .metadata.name, "host-level"] | @tsv'
Pod Security Admission via Namespace Label
kubectl label namespace demo pod-security.kubernetes.io/enforce=baseline --overwrite
kubectl label namespace demo pod-security.kubernetes.io/audit=restricted --overwrite
kubectl label namespace demo pod-security.kubernetes.io/warn=restricted --overwrite
kubectl get namespace demo --show-labels
Check Admission Errors
kubectl apply --dry-run=server -f pod.yaml
kubectl get events -n demo --sort-by=.lastTimestamp
Check Images and Pull Policy
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{.imagePullPolicy}{";"}{end}{"\n"}{end}'
Avoid uncontrolled :latest tags in production-like environments.
23. Resource Management: Requests, Limits, Quotas, and LimitRanges
Show Requests and Limits
kubectl get pod <pod> -n demo -o jsonpath='{.spec.containers[*].resources}'
kubectl describe pod <pod> -n demo
kubectl get pods -n demo -o custom-columns='POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_LIM:.spec.containers[*].resources.limits.memory'
Set Resources
kubectl set resources deployment/web -n demo --requests=cpu=100m,memory=128Mi --limits=cpu=500m,memory=512Mi
kubectl set resources deployment/web -n demo -c app --requests=cpu=200m,memory=256Mi --limits=cpu=1,memory=1Gi
ResourceQuota
kubectl get resourcequota -n demo
kubectl describe resourcequota -n demo
kubectl create quota demo-quota -n demo \
--hard=pods=20,requests.cpu=4,requests.memory=8Gi,limits.cpu=8,limits.memory=16Gi
LimitRange
kubectl get limitrange -n demo
kubectl describe limitrange -n demo
LimitRanges are usually maintained as manifests because the imperative generators are limited.
Check Metrics Server
kubectl top nodes
kubectl top pods -A
kubectl top pods -n demo --containers
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl describe apiservice v1beta1.metrics.k8s.io
If kubectl top does not work, check the Metrics Server and its APIService.
24. Storage: StorageClasses, PVs, PVCs, CSI, and VolumeAttachments
Show StorageClasses
kubectl get storageclass
kubectl describe storageclass <storageclass>
kubectl get storageclass -o yaml
Show PVCs and PVs
kubectl get pvc -A
kubectl get pv
kubectl describe pvc <pvc> -n demo
kubectl describe pv <pv>
Create a PVC
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
namespace: demo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
YAML
Mount a PVC into a Debug Pod
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: pvc-debug
namespace: demo
spec:
restartPolicy: Never
containers:
- name: shell
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: data
YAML
Check CSI
kubectl get csidrivers
kubectl get csinodes
kubectl get volumeattachments
kubectl describe volumeattachment <volumeattachment>
kubectl get pods -A | grep -i csi
Storage Triage
kubectl describe pvc <pvc> -n demo
kubectl describe pod <pod> -n demo
kubectl get events -n demo --sort-by=.lastTimestamp
kubectl get pv,pvc -A
kubectl get storageclass
kubectl get volumeattachments
25. Nodes, Scheduling, Taints, Drain, and Maintenance
Show Nodes
kubectl get nodes
kubectl get nodes -o wide
kubectl describe node <node>
kubectl get node <node> -o yaml
Show Pods on a Node
kubectl get pods -A --field-selector=spec.nodeName=<node> -o wide
Node Labels
kubectl get nodes --show-labels
kubectl label node <node> disktype=ssd
kubectl label node <node> disktype-
Set and Remove Taints
kubectl taint nodes <node> dedicated=gpu:NoSchedule
kubectl taint nodes <node> dedicated=gpu:NoSchedule-
kubectl describe node <node> | grep -i taints -A2
Node cordon and uncordon
kubectl cordon <node>
kubectl get node <node>
kubectl uncordon <node>
cordon prevents new scheduling decisions on the Node. Running Pods remain in place.
Node drain
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=10m
kubectl uncordon <node>
Before drain, check:
kubectl get pdb -A
kubectl get pods -A --field-selector=spec.nodeName=<node> -o wide
kubectl describe node <node>
Analyze Scheduling Problems
kubectl describe pod <pod> -n demo
kubectl get events -n demo --sort-by=.lastTimestamp
kubectl describe node <node>
kubectl get resourcequota,limitrange -n demo
kubectl get pods -A --field-selector=status.phase=Pending
Check Node Pressure
kubectl describe node <node> | grep -E 'Pressure|Allocatable|Capacity' -A5
kubectl get events -A --field-selector=reason=Evicted
kubectl top node <node>
kubectl top pods -A --sort-by=memory
26. Events, Metrics, and Observability
Show Events
kubectl get events -n demo
kubectl get events -n demo --sort-by=.lastTimestamp
kubectl get events -A --sort-by=.lastTimestamp | tail -50
kubectl get events -A --field-selector=type=Warning
Newer events Command, If Supported
kubectl events -n demo
kubectl events -n demo --for pod/<pod>
kubectl events -A --types=Warning
Not every older kubectl version supports kubectl events in the same way.
Show Metrics
kubectl top nodes
kubectl top pods -A
kubectl top pod <pod> -n demo
kubectl top pods -n demo --containers
API Server Health and Readiness
kubectl get --raw /readyz?verbose
kubectl get --raw /livez?verbose
kubectl get --raw /healthz
Cluster Info Dump
kubectl cluster-info
kubectl cluster-info dump
kubectl cluster-info dump --namespaces demo --output-directory ./cluster-dump
cluster-info dump can contain sensitive data. Do not share it without review.
27. kubectl Top-Level Commands at a Glance
| Command | Purpose | Typical use |
|---|---|---|
kubectl annotate |
Set, change, or remove annotations. | Metadata, change cause, controller hints. |
kubectl api-resources |
Show available resource types. | API exploration, CRD checks. |
kubectl api-versions |
Show available API versions. | Check manifest compatibility. |
kubectl apply |
Apply declarative configuration. | Default for GitOps-like changes. |
kubectl attach |
Attach to a running container process. | Interactive diagnosis. |
kubectl auth |
Check authorization and reconcile RBAC. | can-i, whoami, reconcile. |
kubectl autoscale |
Create an HPA. | Start CPU-based autoscaling. |
kubectl certificate |
Manage CertificateSigningRequests. | Approve or deny CSRs. |
kubectl cluster-info |
Show cluster endpoints. | Initial cluster diagnosis. |
kubectl completion |
Generate shell autocomplete. | More productive work. |
kubectl config |
Manage kubeconfig. | Context, cluster, user, namespace. |
kubectl cordon |
Mark a Node unschedulable. | Prepare maintenance. |
kubectl cp |
Copy files between container and local system. | Diagnosis, export small files. |
kubectl create |
Create resources imperatively. | Quick tests, YAML generator. |
kubectl debug |
Start a debug container or Node debug session. | Runtime diagnosis. |
kubectl delete |
Delete resources. | Cleanup, rollback, controller-driven re-creation. |
kubectl describe |
Show detailed description including Events. | Troubleshooting. |
kubectl diff |
Show differences between manifest and cluster. | Review before apply. |
kubectl drain |
Empty a Node for maintenance. | Node patching, hardware maintenance. |
kubectl edit |
Modify a resource interactively. | Emergency change. |
kubectl events |
Show Events, if supported by the version. | Fast Event diagnosis. |
kubectl exec |
Run a command in a container. | Shell, process, and network tests. |
kubectl explain |
Explain API fields. | Manifest development. |
kubectl expose |
Create a Service for a resource. | Fast Service creation. |
kubectl get |
List or retrieve resources. | Default read operation. |
kubectl kustomize |
Render a Kustomization. | Manifest generation. |
kubectl label |
Set, change, or remove labels. | Selectors, organization, policies. |
kubectl logs |
Show container logs. | Application diagnosis. |
kubectl options |
Show global options. | CLI help. |
kubectl patch |
Apply a partial change. | Precise changes to individual fields. |
kubectl plugin |
Show plugins. | CLI extensions. |
kubectl port-forward |
Forward a local port to a Pod or Service. | Local tests. |
kubectl proxy |
Start a local proxy to the Kubernetes API. | API exploration. |
kubectl replace |
Replace a resource. | Full object replacement. |
kubectl rollout |
Manage rollouts. | Status, history, undo, restart. |
kubectl run |
Start a Pod or generate a manifest. | Debugging, quick tests. |
kubectl scale |
Change replica count. | Manual scaling. |
kubectl set |
Set subparts of a resource. | Image, env, resources, ServiceAccount. |
kubectl taint |
Manage Node taints. | Scheduling control. |
kubectl top |
Show metrics. | CPU and memory view. |
kubectl uncordon |
Make a Node schedulable again. | Complete maintenance. |
kubectl version |
Show client and server version. | Check compatibility. |
kubectl wait |
Wait for a condition. | Scripts, CI, rollout validation. |
Help for Every Command
kubectl --help
kubectl get --help
kubectl create --help
kubectl create secret --help
kubectl rollout --help
kubectl rollout history --help
kubectl options
28. kubeadm: Overview
kubeadm is a bootstrapping tool for Kubernetes clusters. It creates a minimally functional cluster, but it does not automatically provide machine provisioning, cloud integration, monitoring, dashboards, or complete add-on installation.
Typical tasks:
| Command | Purpose |
|---|---|
kubeadm init |
Initialize a control-plane Node. |
kubeadm join |
Add a worker or additional control-plane Node to the cluster. |
kubeadm upgrade |
Plan and execute a cluster upgrade. |
kubeadm config |
Show, migrate, validate, and manage kubeadm configuration and images. |
kubeadm token |
Manage bootstrap tokens. |
kubeadm reset |
Remove kubeadm artifacts from a Node. |
kubeadm certs |
Check and renew Kubernetes certificates. |
kubeadm kubeconfig |
Manage kubeconfig files. |
kubeadm version |
Show kubeadm version. |
kubeadm completion |
Generate shell completion. |
kubeadm alpha |
Experimental features; do not use as a stable operations baseline. |
Help
kubeadm --help
kubeadm init --help
kubeadm join --help
kubeadm upgrade --help
kubeadm certs --help
29. kubeadm init
Minimal Control-Plane Start
sudo kubeadm init
This uses defaults and is sufficient for labs, but too imprecise for reproducible setups.
Initialization with Pod CIDR
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
The Pod CIDR must match your CNI. Flannel uses 10.244.0.0/16 in many examples; other CNIs have other defaults.
Initialization with API Server Advertise Address
sudo kubeadm init --apiserver-advertise-address=<ip-address>
This is relevant on hosts with multiple interfaces.
Initialization with Control-Plane Endpoint
sudo kubeadm init \
--control-plane-endpoint=<dns-name-or-load-balancer>:6443 \
--upload-certs
--control-plane-endpoint is important for HA control planes. --upload-certs makes it easier to join additional control-plane Nodes later.
Initialization with Configuration File
sudo kubeadm init --config kubeadm-init.yaml
Example:
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.0.2.10
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.0
controlPlaneEndpoint: k8s-api.example.internal:6443
networking:
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
After init: Set Up kubeconfig
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes
After init: Install CNI
kubectl get nodes
kubectl get pods -n kube-system
kubectl apply -f <cni-manifest.yaml>
kubectl get nodes -w
Without a CNI, Nodes often remain NotReady, and CoreDNS Pods cannot run correctly.
Inspect init with Phases
sudo kubeadm init phase --help
sudo kubeadm init phase preflight --config kubeadm-init.yaml
sudo kubeadm init phase certs all --config kubeadm-init.yaml
sudo kubeadm init phase kubeconfig all --config kubeadm-init.yaml
sudo kubeadm init phase control-plane all --config kubeadm-init.yaml
sudo kubeadm init phase etcd local --config kubeadm-init.yaml
sudo kubeadm init phase upload-config all --config kubeadm-init.yaml
sudo kubeadm init phase bootstrap-token --config kubeadm-init.yaml
sudo kubeadm init phase addon all --config kubeadm-init.yaml
Phases are useful for debugging and special cases. For standard installations, you normally use kubeadm init --config ....
30. kubeadm join
Print the Join Command
On an existing control-plane Node:
kubeadm token create --print-join-command
Join a Worker
On the new worker:
sudo kubeadm join <api-server>:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
Join with Configuration File
sudo kubeadm join --config kubeadm-join.yaml
Example:
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
bootstrapToken:
apiServerEndpoint: k8s-api.example.internal:6443
token: abcdef.0123456789abcdef
caCertHashes:
- sha256:<hash>
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
Join Another Control-Plane Node
sudo kubeadm join <api-server>:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <certificate-key>
Create or renew the certificate key with:
sudo kubeadm init phase upload-certs --upload-certs
Join Phases
sudo kubeadm join phase --help
sudo kubeadm join phase preflight --config kubeadm-join.yaml
sudo kubeadm join phase kubelet-start --config kubeadm-join.yaml
sudo kubeadm join phase control-plane-prepare all --config kubeadm-join.yaml
sudo kubeadm join phase control-plane-join all --config kubeadm-join.yaml
31. kubeadm reset
Reset a Node
sudo kubeadm reset
sudo kubeadm reset -f
kubeadm reset removes local kubeadm artifacts. It is not a full operating system reset.
Reset with CRI Socket
sudo kubeadm reset -f --cri-socket unix:///var/run/containerd/containerd.sock
Check After Reset
sudo systemctl status kubelet
sudo ls -la /etc/kubernetes/
sudo ls -la /var/lib/kubelet/
sudo iptables-save | grep -i kube || true
ip link
Depending on the CNI, you may need to clean up network interfaces, iptables/nftables rules, or CNI directories separately.
Carefully Remove CNI Leftovers
sudo rm -rf /etc/cni/net.d
sudo rm -rf /var/lib/cni
Run this only if the Node is really being removed from the cluster or rebuilt.
32. kubeadm upgrade
Check Current Versions
kubectl get nodes
kubectl version
kubeadm version
Show Upgrade Plan
sudo kubeadm upgrade plan
Upgrade the Control Plane
sudo kubeadm upgrade apply v1.36.0
Adjust the version to your target version. In practice, you install the matching kubeadm package version beforehand.
Node Upgrade Step
On additional control-plane Nodes and workers:
sudo kubeadm upgrade node
Show Differences
sudo kubeadm upgrade diff v1.36.0
Typical Per-Node Flow
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
# update kubeadm/kubelet/kubectl through the package manager
sudo kubeadm upgrade node
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon <node>
kubectl get nodes
For control-plane Nodes, also check the API Server, Controller Manager, Scheduler, and etcd.
33. kubeadm config and Images
Print Defaults
kubeadm config print init-defaults
kubeadm config print join-defaults
Migrate Configuration
kubeadm config migrate --old-config old.yaml --new-config new.yaml
Validate Configuration
kubeadm config validate --config kubeadm-init.yaml
Read Cluster Configuration from ConfigMap
kubectl get configmap kubeadm-config -n kube-system -o yaml
kubectl get configmap kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}'
List and Pre-Pull Images
kubeadm config images list
kubeadm config images list --kubernetes-version v1.36.0
sudo kubeadm config images pull
sudo kubeadm config images pull --kubernetes-version v1.36.0
sudo kubeadm config images pull --cri-socket unix:///var/run/containerd/containerd.sock
This is important for air-gapped environments or controlled registry mirrors.
34. kubeadm token, certs, and kubeconfig
Show and Create Bootstrap Tokens
kubeadm token list
kubeadm token generate
kubeadm token create
kubeadm token create --ttl 2h
kubeadm token create --print-join-command
kubeadm token delete <token-id>
Check Certificate Expiration
sudo kubeadm certs check-expiration
Renew Certificates
sudo kubeadm certs renew all
sudo kubeadm certs renew apiserver
sudo kubeadm certs renew apiserver-kubelet-client
sudo kubeadm certs renew front-proxy-client
sudo kubeadm certs renew etcd-server
sudo kubeadm certs renew etcd-peer
sudo kubeadm certs renew etcd-healthcheck-client
sudo kubeadm certs renew apiserver-etcd-client
After renewing certificates, you must restart or refresh static Pods or relevant components, often by restarting kubelet or rewriting/moving the manifests.
Generate Certificate Key
sudo kubeadm certs certificate-key
sudo kubeadm init phase upload-certs --upload-certs
Generate CSRs
sudo kubeadm certs generate-csr
Manage kubeconfig Files
sudo kubeadm kubeconfig --help
sudo kubeadm kubeconfig user --client-name <name>
sudo kubeadm init phase kubeconfig admin
sudo kubeadm init phase kubeconfig kubelet
sudo kubeadm init phase kubeconfig controller-manager
sudo kubeadm init phase kubeconfig scheduler
Typical kubeadm kubeconfigs are located under:
ls -la /etc/kubernetes/*.conf
35. kubeadm phases, completion, version, and alpha
Version
kubeadm version
kubeadm version -o yaml
kubeadm version -o json
Completion
source <(kubeadm completion bash)
kubeadm completion zsh
kubeadm completion fish
Show Phases
kubeadm init phase --help
kubeadm join phase --help
kubeadm reset phase --help
Alpha Features
kubeadm alpha --help
Alpha features can change or disappear. Do not use them as a stable foundation for production runbooks.
36. kubelet and systemd
kubelet is the Node agent. In kubeadm setups, it usually runs as a systemd service and starts static Pods from /etc/kubernetes/manifests.
Check the Service
sudo systemctl status kubelet
sudo systemctl is-enabled kubelet
sudo systemctl cat kubelet
sudo journalctl -u kubelet -n 200 --no-pager
sudo journalctl -u kubelet -f
Control the Service
sudo systemctl daemon-reload
sudo systemctl restart kubelet
sudo systemctl stop kubelet
sudo systemctl start kubelet
Check kubelet Configuration
sudo cat /var/lib/kubelet/config.yaml
sudo cat /var/lib/kubelet/kubeadm-flags.env
sudo cat /etc/default/kubelet 2>/dev/null || true
sudo cat /etc/sysconfig/kubelet 2>/dev/null || true
Call kubelet Directly
kubelet --help
kubelet --version
Direct kubelet starts are uncommon in operations. You normally inspect and control kubelet through systemd.
Check Common kubelet Errors
sudo journalctl -u kubelet -n 300 --no-pager | grep -Ei 'error|failed|certificate|cni|runtime|eviction'
crictl info
crictl ps -a
kubectl describe node <node>
37. Control-Plane Components: kube-apiserver, Scheduler, Controller Manager
In kubeadm clusters, control-plane components usually run as static Pods.
Show Static Pod Manifests
sudo ls -la /etc/kubernetes/manifests
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml
sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml
sudo cat /etc/kubernetes/manifests/etcd.yaml
Check Control-Plane Pods
kubectl get pods -n kube-system -o wide
kubectl describe pod -n kube-system kube-apiserver-<node>
kubectl logs -n kube-system kube-apiserver-<node> --tail=100
kubectl logs -n kube-system kube-controller-manager-<node> --tail=100
kubectl logs -n kube-system kube-scheduler-<node> --tail=100
kube-apiserver
kube-apiserver --help
kubectl get --raw /readyz?verbose
kubectl get --raw /livez?verbose
kubectl get --raw /metrics | head
Important API Server topics include certificates, Admission Controllers, authentication, authorization, audit logging, etcd connectivity, and feature gates.
kube-controller-manager
kube-controller-manager --help
kubectl logs -n kube-system kube-controller-manager-<node> --tail=200
The Controller Manager contains central control loops that reconcile actual state toward desired state.
kube-scheduler
kube-scheduler --help
kubectl logs -n kube-system kube-scheduler-<node> --tail=200
The Scheduler chooses Nodes for Pods. Scheduling errors are often found in Pod Events, not only in Scheduler logs.
38. kube-proxy
kube-proxy implements Service networking rules, depending on the mode through iptables, IPVS, nftables, or other distribution-specific mechanisms.
Check kube-proxy
kubectl get daemonset kube-proxy -n kube-system
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=100
kubectl get configmap kube-proxy -n kube-system -o yaml
Direct Binary
kube-proxy --help
kube-proxy --version
Check Service Rules on a Node
sudo iptables-save | grep KUBE-SVC | head
sudo iptables-save | grep KUBE-SEP | head
sudo ipvsadm -Ln 2>/dev/null | head || true
sudo nft list ruleset 2>/dev/null | grep -i kube | head || true
Which commands work depends on the kube-proxy mode and operating system.
39. Container Runtime: crictl, ctr, nerdctl, and Docker
crictl: Standard Tool for CRI Debugging
crictl talks to the Kubernetes Container Runtime Interface API. For Node debugging, crictl is usually the most appropriate tool.
Configure Endpoint
sudo crictl config runtime-endpoint unix:///var/run/containerd/containerd.sock
sudo crictl config image-endpoint unix:///var/run/containerd/containerd.sock
sudo cat /etc/crictl.yaml
For CRI-O:
sudo crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps
Runtime Info
sudo crictl info
sudo crictl version
Show Pods and Containers
sudo crictl pods
sudo crictl pods -a
sudo crictl ps
sudo crictl ps -a
sudo crictl inspect <container-id>
sudo crictl inspectp <pod-id>
Logs and Exec
sudo crictl logs <container-id>
sudo crictl logs --tail=100 <container-id>
sudo crictl exec -it <container-id> sh
Images
sudo crictl images
sudo crictl pull <image>
sudo crictl rmi <image-id>
sudo crictl inspecti <image-id>
Stop and Remove Containers
sudo crictl stop <container-id>
sudo crictl rm <container-id>
sudo crictl rmp <pod-id>
Use stop/remove on Kubernetes containers only for diagnosis. kubelet re-creates managed containers.
ctr: containerd-Native Tool
ctr talks directly to containerd and operates at a lower level than crictl.
sudo ctr version
sudo ctr namespaces list
sudo ctr -n k8s.io containers list
sudo ctr -n k8s.io tasks list
sudo ctr -n k8s.io images list
sudo ctr -n k8s.io snapshots list
sudo ctr -n k8s.io content ls
Kubernetes objects in containerd are usually in the k8s.io namespace.
nerdctl
nerdctl provides Docker-like usage for containerd. For Kubernetes runtime debugging, crictl usually remains more correct because it shows the CRI view.
sudo nerdctl version
sudo nerdctl -n k8s.io ps
sudo nerdctl -n k8s.io images
sudo nerdctl -n k8s.io logs <container>
sudo nerdctl -n k8s.io inspect <container>
Docker
In modern Kubernetes clusters, Docker is no longer the direct Kubernetes runtime. Docker commands are still relevant when you build images locally.
docker build -t <image> .
docker tag <image> <registry>/<image>:<tag>
docker push <registry>/<image>:<tag>
docker run --rm -it <image> sh
docker logs <container>
docker inspect <container>
For containers started by kubelet through containerd or CRI-O, use crictl ps, not docker ps.
40. etcd, etcdctl, and etcdutl
etcd stores the cluster state. Work on etcd is critical. Before changes, you need backups, a change window, and a restore plan.
Check etcd Pod in kubeadm
kubectl get pods -n kube-system -l component=etcd -o wide
kubectl logs -n kube-system etcd-<node> --tail=100
sudo cat /etc/kubernetes/manifests/etcd.yaml
etcdctl Environment for kubeadm
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/healthcheck-client.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/healthcheck-client.key
Endpoint Health and Status
sudo -E etcdctl endpoint health
sudo -E etcdctl endpoint status -w table
sudo -E etcdctl member list -w table
Create Snapshot
sudo -E etcdctl snapshot save /var/backups/etcd-snapshot.db
sudo -E etcdctl snapshot status /var/backups/etcd-snapshot.db -w table
Check Snapshot with etcdutl
etcdutl snapshot status /var/backups/etcd-snapshot.db -w table
Restore Snapshot
etcdutl snapshot restore /var/backups/etcd-snapshot.db --data-dir /var/lib/etcd-restore
For Kubernetes restore scenarios, revision bumps are important so informer caches do not operate against revisions that move backward:
etcdutl snapshot restore /var/backups/etcd-snapshot.db \
--bump-revision 1000000000 \
--mark-compacted \
--data-dir /var/lib/etcd-restore
Inspect Keys
sudo -E etcdctl get / --prefix --keys-only | head
sudo -E etcdctl get /registry/namespaces --prefix --keys-only
In normal operation, read Kubernetes state through the API, not directly from etcd. Direct etcd queries are diagnostic and disaster-recovery tools.
41. Helm
Helm manages Kubernetes applications as charts and releases.
Repositories
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo list
helm repo update
helm search repo nginx
helm repo remove bitnami
Inspect a Chart
helm show chart bitnami/nginx
helm show values bitnami/nginx
helm show readme bitnami/nginx
helm show crds <chart>
helm pull bitnami/nginx --untar
Check and Render a Chart
helm lint ./chart
helm template web ./chart -f values.yaml
helm template web ./chart -f values.yaml --debug
helm template web ./chart -n demo --include-crds
Install and Upgrade
helm install web ./chart -n demo --create-namespace
helm upgrade web ./chart -n demo -f values.yaml
helm upgrade --install web ./chart -n demo --create-namespace -f values.yaml
helm upgrade --install web ./chart -n demo --atomic --wait --timeout 5m
helm upgrade --install web ./chart -n demo --dry-run --debug
--atomic rolls back on errors. --wait waits for Kubernetes readiness conditions as far as Helm can evaluate them.
Show Releases
helm list -n demo
helm list -A
helm status web -n demo
helm history web -n demo
Retrieve Release Content
helm get values web -n demo
helm get values web -n demo --all
helm get manifest web -n demo
helm get hooks web -n demo
helm get notes web -n demo
helm get all web -n demo
Rollback and Uninstall
helm rollback web 1 -n demo
helm uninstall web -n demo
helm uninstall web -n demo --keep-history
Dependencies
helm dependency list ./chart
helm dependency update ./chart
helm dependency build ./chart
Package and OCI Registry
helm package ./chart
helm registry login <registry>
helm push ./chart-0.1.0.tgz oci://<registry>/<repo>
helm pull oci://<registry>/<repo>/<chart> --version 0.1.0
Helm Plugins
helm plugin list
helm plugin install <plugin-url>
helm plugin update <plugin-name>
helm plugin uninstall <plugin-name>
42. Kustomize
Kustomize renders Kubernetes manifests from bases and overlays.
Kustomize Integrated into kubectl
kubectl kustomize overlays/dev
kubectl apply -k overlays/dev
kubectl diff -k overlays/dev
kubectl delete -k overlays/dev
Standalone kustomize Binary
kustomize build overlays/dev
kustomize build overlays/prod > rendered.yaml
kustomize edit set image app=ghcr.io/example/app:1.2.3
kustomize edit set namespace demo
kustomize edit add resource deployment.yaml
Check a Typical Overlay
find overlays/dev -maxdepth 2 -type f -name 'kustomization.yaml' -print
kubectl kustomize overlays/dev | kubectl apply --dry-run=server -f -
kubectl kustomize overlays/dev | kubectl diff -f -
ConfigMap and Secret Generators
kustomize edit add configmap app-config --from-literal=LOG_LEVEL=info
kustomize edit add secret app-secret --from-literal=password=dummy
Again: do not store real Secrets in Git.
43. Local and Lightweight Clusters: kind, minikube, k3d, k3s, k0s, and Talos
These tools are not Kubernetes core, but they are common in workshops and labs.
kind
kind version
kind create cluster
kind create cluster --name workshop
kind get clusters
kind get kubeconfig --name workshop
kind load docker-image <image> --name workshop
kind export logs ./kind-logs --name workshop
kind delete cluster --name workshop
minikube
minikube version
minikube start
minikube start --driver=docker
minikube status
minikube kubectl -- get pods -A
minikube addons list
minikube addons enable ingress
minikube service <service> -n <namespace>
minikube tunnel
minikube logs
minikube delete
k3d
k3d version
k3d cluster create workshop
k3d cluster list
k3d kubeconfig get workshop
k3d image import <image> -c workshop
k3d cluster delete workshop
k3s
sudo systemctl status k3s
sudo journalctl -u k3s -f
sudo k3s kubectl get nodes
sudo cat /etc/rancher/k3s/k3s.yaml
k0s
k0s version
sudo k0s status
sudo k0s kubectl get nodes
sudo k0s kubeconfig admin
sudo systemctl status k0scontroller
sudo systemctl status k0sworker
Talos
talosctl version
talosctl config endpoint <node-ip>
talosctl config node <node-ip>
talosctl health
talosctl kubeconfig .
talosctl service kubelet
talosctl logs kubelet
talosctl dmesg
talosctl dashboard
Talos does not use a classic SSH or shell model. You work through talosctl.
44. Cloud Clusters: EKS, GKE, and AKS
Cloud CLIs often create kubeconfig entries. After that, you work with kubectl again.
AWS EKS
aws eks list-clusters --region <region>
aws eks describe-cluster --name <cluster> --region <region>
aws eks update-kubeconfig --name <cluster> --region <region>
kubectl get nodes
Google GKE
gcloud container clusters list --region <region>
gcloud container clusters get-credentials <cluster> --region <region> --project <project>
kubectl get nodes
Azure AKS
az aks list -o table
az aks get-credentials --resource-group <resource-group> --name <cluster>
az aks get-credentials --resource-group <resource-group> --name <cluster> --overwrite-existing
kubectl get nodes
Cloud-specific permissions, network models, LoadBalancers, StorageClasses, and managed add-ons differ significantly. Always check the provider documentation.
45. CNI, Service Mesh, and Add-on Commands
These commands depend on installed add-ons.
Cilium
cilium status
cilium version
cilium connectivity test
cilium sysdump
cilium config view
cilium policy get
hubble status
hubble observe
hubble observe --namespace demo
Calico
calicoctl version
calicoctl get nodes
calicoctl get ippools
calicoctl get networkpolicy -A
calicoctl get globalnetworkpolicy
calicoctl get felixconfiguration default -o yaml
Istio
istioctl version
istioctl proxy-status
istioctl analyze
istioctl x describe pod <pod> -n <namespace>
istioctl proxy-config clusters <pod> -n <namespace>
istioctl proxy-config listeners <pod> -n <namespace>
istioctl proxy-config routes <pod> -n <namespace>
istioctl proxy-config endpoints <pod> -n <namespace>
cert-manager
kubectl get certificates -A
kubectl get certificaterequests -A
kubectl get issuers -A
kubectl get clusterissuers
kubectl describe certificate <certificate> -n <namespace>
kubectl describe certificaterequest <request> -n <namespace>
kubectl get challenges,orders -A
Argo CD CLI and kubectl View
argocd version
argocd login <server>
argocd app list
argocd app get <app>
argocd app sync <app>
argocd app diff <app>
argocd app history <app>
kubectl get applications -n argocd
kubectl describe application <app> -n argocd
Flux CLI
flux version
flux check
flux get sources all -A
flux get kustomizations -A
flux get helmreleases -A
flux reconcile kustomization <name> -n <namespace>
flux reconcile source git <name> -n <namespace>
Krew and kubectl Plugins
kubectl plugin list
kubectl krew version
kubectl krew search
kubectl krew install ctx
kubectl krew install ns
kubectl krew install neat
kubectl krew update
kubectl krew upgrade
Plugins are useful, but they are not available in every cluster or on every admin machine. Runbooks should remain understandable without non-essential plugins.
46. Linux Helper Commands for Kubernetes Operations
These commands run on Nodes or admin machines.
systemd and Logs
sudo systemctl status kubelet
sudo systemctl status containerd
sudo systemctl status crio
sudo journalctl -u kubelet -n 200 --no-pager
sudo journalctl -u containerd -n 200 --no-pager
sudo journalctl -u crio -n 200 --no-pager
Network
ip addr
ip route
ss -lntup
ss -antp | grep 6443
sudo iptables-save | grep -i kube
sudo nft list ruleset | grep -i kube
sudo ipvsadm -Ln
conntrack -L 2>/dev/null | head
DNS
resolvectl status 2>/dev/null || true
cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local <cluster-dns-ip>
dig kubernetes.default.svc.cluster.local @<cluster-dns-ip>
Files and Mounts
df -h
df -ih
mount | grep kubelet
findmnt | grep kubelet
sudo du -sh /var/lib/kubelet/* 2>/dev/null | sort -h
sudo du -sh /var/lib/containerd/* 2>/dev/null | sort -h
Certificates
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | less
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subject -issuer
Processes
ps aux | grep kubelet
ps aux | grep containerd
ps aux | grep kube-apiserver
sudo crictl ps -a
47. JSONPath, jq, yq, custom-columns, and Templates
JSONPath Examples
kubectl get pods -n demo -o jsonpath='{.items[*].metadata.name}'
kubectl get pod <pod> -n demo -o jsonpath='{.status.phase}'
kubectl get pod <pod> -n demo -o jsonpath='{.spec.nodeName}'
kubectl get secret app-secret -n demo -o jsonpath='{.data.password}' | base64 -d; echo
List with Line Breaks
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'
custom-columns
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase'
kubectl get nodes -o custom-columns='NODE:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion'
jq
kubectl get pods -A -o json | jq '.items[] | {namespace: .metadata.namespace, name: .metadata.name, phase: .status.phase}'
kubectl get pods -A -o json | jq -r '.items[] | [.metadata.namespace, .metadata.name, .status.phase] | @tsv'
yq
yq '.spec.template.spec.containers[].image' deployment.yaml
kubectl get deployment web -n demo -o yaml | yq '.spec.template.spec.containers[].image'
Go Template
kubectl get pods -A -o go-template='{{range .items}}{{.metadata.namespace}}{{"\t"}}{{.metadata.name}}{{"\n"}}{{end}}'
48. Troubleshooting Recipes
Fast Cluster Triage
kubectl config current-context
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl get componentstatuses 2>/dev/null || true
kubectl get --raw /readyz?verbose
Pod Pending
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get nodes -o wide
kubectl describe nodes | grep -E 'Taints:|Allocatable:|Allocated resources:' -A10
kubectl get resourcequota,limitrange -n <namespace>
Typical causes: insufficient resources, taints without tolerations, NodeSelector/Affinity mismatch, unbound PVC, quota reached.
CrashLoopBackOff
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers --previous
kubectl logs <pod> -n <namespace> --all-containers --tail=200
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses}'
Typical causes: process exits, wrong configuration, missing Secret, overly aggressive probe, permission problem, image does not start.
ImagePullBackOff or ErrImagePull
kubectl describe pod <pod> -n <namespace>
kubectl get secret -n <namespace>
kubectl get serviceaccount -n <namespace> <serviceaccount> -o yaml
kubectl get events -n <namespace> --sort-by=.lastTimestamp
Typical causes: wrong image name, missing tag, registry unreachable, missing ImagePullSecret, rate limit.
Service Without Endpoints
kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpoints <service> -n <namespace> -o wide
kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service>
kubectl get pods -n <namespace> --show-labels
kubectl describe pod -n <namespace> -l <selector>
Typical causes: wrong selector, Pods not Ready, wrong port name, Pods in a different namespace.
Broken DNS
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
kubectl get configmap coredns -n kube-system -o yaml
kubectl run dns-test -n <namespace> --rm -it --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local
Ingress Does Not Route
kubectl get ingress -A
kubectl describe ingress <ingress> -n <namespace>
kubectl get ingressclass
kubectl get svc,ep,endpointslice -n <namespace>
kubectl get pods -A | grep -i ingress
kubectl logs -n <ingress-namespace> <ingress-controller-pod> --tail=200
Gateway Route Does Not Work
kubectl get gatewayclasses,gateway,httproute -A
kubectl describe gateway <gateway> -n <namespace>
kubectl describe httproute <route> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
PVC Pending
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass
kubectl get pv
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get pods -A | grep -i csi
Node NotReady
kubectl describe node <node>
kubectl get pods -A --field-selector=spec.nodeName=<node> -o wide
sudo systemctl status kubelet
sudo journalctl -u kubelet -n 300 --no-pager
sudo crictl info
sudo crictl ps -a
RBAC Forbidden
kubectl auth can-i <verb> <resource> -n <namespace>
kubectl auth can-i <verb> <resource> -n <namespace> --as system:serviceaccount:<namespace>:<serviceaccount>
kubectl get role,rolebinding -n <namespace>
kubectl get clusterrole,clusterrolebinding | grep <name>
OOMKilled
kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
kubectl top pod <pod> -n <namespace> --containers
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'
Rollout Stuck
kubectl rollout status deployment/<deployment> -n <namespace> --timeout=60s
kubectl describe deployment <deployment> -n <namespace>
kubectl get rs,pods -n <namespace> -l app=<app>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs deployment/<deployment> -n <namespace> --all-containers --tail=100
API Server Unreachable
kubectl cluster-info
kubectl get --raw /readyz?verbose
curl -k https://<api-server>:6443/readyz
sudo crictl ps -a | grep kube-apiserver
sudo crictl logs <apiserver-container-id>
sudo journalctl -u kubelet -n 300 --no-pager
49. Operations Checklists
Before a Change
kubectl config current-context
kubectl get namespace <namespace>
kubectl auth can-i patch deployments -n <namespace>
kubectl diff -f manifest.yaml
kubectl apply --dry-run=server -f manifest.yaml
Also check:
- Is the cluster correct?
- Is the namespace correct?
- Is there a rollback path?
- Is there a backup if data is affected?
- Are real Secrets excluded?
- Are PDBs and capacity sufficient?
After a Change
kubectl rollout status deployment/<deployment> -n <namespace> --timeout=120s
kubectl get pods -n <namespace> -o wide
kubectl get events -n <namespace> --sort-by=.lastTimestamp | tail -50
kubectl logs deployment/<deployment> -n <namespace> --all-containers --tail=100
kubectl get svc,ep,endpointslice -n <namespace>
Before Node Maintenance
kubectl get node <node> -o wide
kubectl get pods -A --field-selector=spec.nodeName=<node> -o wide
kubectl get pdb -A
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=10m
After Node Maintenance
sudo systemctl restart kubelet
kubectl uncordon <node>
kubectl get node <node> -o wide
kubectl get pods -A --field-selector=spec.nodeName=<node> -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -50
Before kubeadm Upgrade
kubectl get nodes
kubectl get pods -A
kubectl get pdb -A
sudo kubeadm upgrade plan
sudo kubeadm certs check-expiration
sudo -E etcdctl endpoint health
sudo -E etcdctl snapshot save /var/backups/etcd-before-upgrade.db
After kubeadm Upgrade
kubectl get nodes -o wide
kubectl get pods -A
kubectl get --raw /readyz?verbose
kubectl top nodes
kubectl get events -A --sort-by=.lastTimestamp | tail -100
50. Sources and Further References
- Kubernetes kubectl Reference: https://kubernetes.io/docs/reference/kubectl/
- Kubernetes kubectl Quick Reference: https://kubernetes.io/docs/reference/kubectl/quick-reference/
- Kubernetes kubeadm Reference: https://kubernetes.io/docs/reference/setup-tools/kubeadm/
- Kubernetes kubelet Reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
- Kubernetes Component Tools Reference: https://kubernetes.io/docs/reference/command-line-tools-reference/
- Kubernetes crictl Debugging: https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/
- Helm Commands: https://helm.sh/docs/helm/
- etcd Disaster Recovery: https://etcd.io/docs/v3.5/op-guide/recovery/