1. Objective and Scope

This document describes the full subject area of security in Kubernetes. It is intended for workshops, internal platform standards, security reviews, operational handovers, and architecture decisions.

After working through this document, you should be able to:

• structure Kubernetes security professionally;
• assess risks by environment;
• define secure defaults for workloads;
• implement API, RBAC, pod, network, secret, node, and supply-chain controls;
• classify security requirements for kind, minikube, k3d, WSL, local VMs, remote VMs, bare metal, kubeadm clusters, k3s, RKE2, k0s, Talos, EKS, AKS, GKE, OKE, DOKS, LKE, and OpenShift;
• triage security findings;
• handle incidents reproducibly;
• translate compliance and audit requirements into technical controls.

1.1. What This Document Covers

1.2. What This Document Does Not Replace

This document does not replace:

• legal compliance assessment;
• an organization-wide risk analysis;
• vendor documentation for your provider;
• penetration tests;
• emergency exercises;
• production approvals by security, privacy, or governance teams.

It provides the technical foundation needed to prepare these processes properly.

1.3. Terms

2. Security Guiding Principles

Kubernetes security is not a single feature. It emerges from layered controls, clear responsibilities, and repeatability.

2.1. Principles

1. Declarative instead of manual. Security controls belong in Git, IaC, Helm, Kustomize, or policy engines.
2. Least privilege everywhere. Users, ServiceAccounts, nodes, cloud roles, registry access, and CI/CD tokens receive only the rights they need.
3. Default deny before allow lists. Network, RBAC, admission, and egress should start with restrictive defaults.
4. Identity instead of static secrets. Prefer OIDC, Workload Identity, Pod Identity, Managed Identity, or short-lived tokens.
5. Immutable instead of repair. Replace compromised nodes and unsafe images instead of manually cleaning them up.
6. Admission catches issues early. Anything that must not run should ideally never be stored in etcd.
7. Observability is a security control. Without audit logs, metrics, runtime events, and centralized logs, security is blind.
8. Restore is part of security. Backups without a tested restore are not a reliable control.
9. The supply chain starts before Kubernetes. Images, dependencies, build runners, registries, and GitOps controllers are part of the threat model.
10. Environment determines risk. A local kind lab needs different controls from a regulated multi-tenant production cluster.

2.2. Minimal Security Standard for Production-like Clusters

2.3. Security Maturity Levels

3. Quick Start: Security Baseline

This section provides an immediately usable baseline for labs and production-like tests. For real production, adapt every control to your organization, regulatory context, network architecture, and provider constraints.

3.1. Namespace with Pod Security Admission

apiVersion: v1
kind: Namespace
metadata:
  name: app-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Validation:

kubectl apply -f namespace-app-prod.yaml
kubectl get ns app-prod --show-labels

3.2. Default Deny for Ingress and Egress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: app-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

This policy also blocks DNS egress unless DNS is explicitly allowed.

3.3. Allow DNS Egress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: app-prod
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Adjust the labels to your CoreDNS installation. Some distributions use different labels.

3.4. Secure ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-runtime
  namespace: app-prod
automountServiceAccountToken: false

If your application does not need the Kubernetes API, do not mount a token. If it does need API access, grant exactly the required permissions and enable token mounting only for that workload.

3.5. Minimal RBAC for Read-Only Pod Access in the Namespace

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: app-prod
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-app-runtime
  namespace: app-prod
subjects:
  - kind: ServiceAccount
    name: app-runtime
    namespace: app-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader

3.6. Hardened Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: app-prod
  labels:
    app.kubernetes.io/name: demo-app
    app.kubernetes.io/part-of: workshop
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: demo-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: demo-app
        app.kubernetes.io/part-of: workshop
    spec:
      serviceAccountName: app-runtime
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: ghcr.io/example/demo-app@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 10001
            runAsGroup: 10001
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 20
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

3.7. Baseline Validation

kubectl auth can-i list pods -n app-prod --as system:serviceaccount:app-prod:app-runtime
kubectl auth can-i get secrets -n app-prod --as system:serviceaccount:app-prod:app-runtime
kubectl get networkpolicy -n app-prod
kubectl get pods -n app-prod -o wide
kubectl describe pod -n app-prod -l app.kubernetes.io/name=demo-app

Expected result:

• the ServiceAccount can read Pods but cannot read Secrets;
• Pods run as non-root, without privilege escalation, and with RuntimeDefault seccomp;
• networking is closed by default;
• exceptions are explicitly documented.

4. Threat Model

Kubernetes has several attack surfaces. A useful security model considers not just Pods, but the full delivery and operations path.

4.1. Primary Attack Surfaces

4.2. Attacker Profiles

4.3. Typical Attack Chains

flowchart LR
    A[Public App CVE] --> B[Shell in Pod]
    B --> C[ServiceAccount Token]
    C --> D[Check RBAC]
    D --> E{Rights too broad?}
    E -- yes --> F[Read Secrets or create Pods]
    E -- no --> G[Explore node or network]
    F --> H[Cloud credentials or DB access]
    G --> I[Egress to C2 or lateral movement]

flowchart LR
    A[CI Runner compromised] --> B[Image manipulated]
    B --> C[Registry Push]
    C --> D[GitOps Rollout]
    D --> E[Pod starts in cluster]
    E --> F[Runtime Detection or Admission?]
    F -- no control --> G[Persistence and data access]
    F -- control active --> H[Deployment blocked or alert]

4.4. Threats by STRIDE

5. Defense-in-Depth Architecture

Security in Kubernetes is layered. A single policy is never enough.

5.1. Overview

flowchart TB
    subgraph Supply[Supply Chain]
      Code[Code Review]
      Build[Build Runner]
      Scan[Scan and SBOM]
      Sign[Signature and Provenance]
      Registry[Registry]
    end

    subgraph Access[Access]
      User[Users / Groups]
      OIDC[OIDC / SSO / MFA]
      Kubeconfig[kubeconfig / Exec Plugin]
    end

    subgraph API[Kubernetes API]
      APIServer[API Server]
      Authn[Authentication]
      Authz[RBAC / Webhook Authz]
      Admission[Admission / Policy]
      Audit[Audit Logs]
      Etcd[etcd]
    end

    subgraph Runtime[Runtime]
      Node[Node]
      Kubelet[kubelet]
      RuntimeEngine[containerd / CRI-O]
      Pod[Pod]
      Net[NetworkPolicy / CNI]
      Detect[Runtime Detection]
    end

    Code --> Build --> Scan --> Sign --> Registry
    User --> OIDC --> Kubeconfig --> APIServer
    Registry --> Admission
    APIServer --> Authn --> Authz --> Admission --> Etcd
    APIServer --> Audit
    Etcd --> Kubelet
    Kubelet --> RuntimeEngine --> Pod
    Pod --> Net
    RuntimeEngine --> Detect
    Audit --> Detect

5.2. API Request Path

sequenceDiagram
    participant Client as kubectl / Controller / GitOps
    participant API as kube-apiserver
    participant Authn as Authentication
    participant Authz as Authorization
    participant Admission as Admission
    participant Etcd as etcd
    participant Audit as Audit Backend

    Client->>API: Request
    API->>Audit: RequestReceived
    API->>Authn: Who are you?
    Authn-->>API: user / groups / ServiceAccount
    API->>Authz: Are you allowed?
    Authz-->>API: allow / deny
    API->>Admission: validate / mutate
    Admission-->>API: allow / deny / patch
    API->>Etcd: persist
    API->>Audit: ResponseComplete
    API-->>Client: response

5.3. Control Matrix

6. Environments and Responsibilities

Security controls vary significantly by environment. Always clarify who controls which layer.

6.1. Responsibility Model

6.2. Decision Diagram

flowchart TD
    A[Which environment?] --> B{Learning or CI only?}
    B -- yes --> C[kind, minikube, k3d]
    B -- no --> D{Production-like operation?}
    D -- local --> E[VM, bare metal, kubeadm, k3s, RKE2, Talos]
    D -- cloud --> F[EKS, AKS, GKE, OKE, DOKS, LKE]
    D -- Enterprise PaaS --> G[OpenShift]
    C --> H[No real secrets, protect host, test policies]
    E --> I[Harden OS, runtime, API, etcd, CNI, storage]
    F --> J[Provider IAM, private endpoints, workload identity, logs]
    G --> K[SCC, OAuth, operators, projects, ClusterRoles]

6.3. Security Exceptions by Environment

7. API Security

The API Server is the central control plane. Almost every security decision eventually reaches the Kubernetes API.

7.1. Risks

• kubeconfig files with admin rights are shared.
• API endpoints are publicly reachable.
• old certificates or tokens do not expire.
• users receive cluster-admin because RBAC design is missing.
• audit logs are missing or not evaluated.
• admission webhooks are operated insecurely or allow everything when they fail.

7.2. Securing API Access

7.3. Handling kubeconfig Securely

kubectl config get-contexts
kubectl config view --minify
kubectl config current-context

Review regularly:

find ~/.kube -maxdepth 2 -type f -print -exec ls -l {} \;

Minimum rules:

• ~/.kube/config should be readable only by your user.
• kubeconfigs do not belong in Git.
• kubeconfigs do not belong in tickets, chat, wikis, or training material.
• training uses demo clusters and demo credentials.
• CI/CD uses short-lived OIDC flows or dedicated ServiceAccounts with minimal permissions.

7.4. API Server Flags in Self-Managed Clusters

In kubeadm clusters, API Server parameters usually live in the static Pod manifest:

sudo grep -n "audit\|authorization\|admission\|encryption" /etc/kubernetes/manifests/kube-apiserver.yaml

Important topics:

7.5. API Security Checks

kubectl version
kubectl cluster-info
kubectl auth whoami
kubectl auth can-i --list
kubectl get --raw /readyz?verbose
kubectl get --raw /livez?verbose

For production environments, also check:

kubectl get clusterrolebinding -o wide
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding
kubectl get mutatingadmissionpolicy,mutatingadmissionpolicybinding

8. Authentication, Authorization, and Admission

Every API request goes through multiple stages.

flowchart LR
    Request[API Request] --> Authn[Authentication]
    Authn --> Authz[Authorization]
    Authz --> Admission[Admission]
    Admission --> Etcd[Persistence in etcd]
    Admission --> Audit[Audit]

8.1. Authentication

Kubernetes has no native User API for humans. User identities come from external mechanisms.

Recommendations:

• humans use SSO/OIDC with MFA;
• machines use short-lived, purpose-bound tokens;
• legacy token Secrets are avoided;
• certificate lifetimes are monitored.

8.2. Authorization

The most common authorization mechanism is RBAC. Kubernetes can use other modes, but RBAC is the primary control in most platforms.

Authorization questions:

kubectl auth can-i get pods -n demo
kubectl auth can-i create deployments -n demo
kubectl auth can-i get secrets -A
kubectl auth can-i '*' '*' --all-namespaces
kubectl auth can-i get pods --as alice@example.com -n demo
kubectl auth can-i get secrets --as system:serviceaccount:demo:app -n demo

8.3. Admission

Admission controllers run after authentication and authorization but before persistence in etcd.

8.4. Handling Admission Failures Safely

Webhook outages are an operational and security risk.

8.5. Admission Review Commands

kubectl get validatingwebhookconfiguration -o wide
kubectl get mutatingwebhookconfiguration -o wide
kubectl describe validatingwebhookconfiguration <name>
kubectl describe mutatingwebhookconfiguration <name>
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding
kubectl get mutatingadmissionpolicy,mutatingadmissionpolicybinding
kubectl get events -A --sort-by=.lastTimestamp | grep -i admission

9. RBAC and Least Privilege

RBAC is one of the most important Kubernetes controls. Misconfigurations can lead directly to privilege escalation.

9.1. RBAC Objects

9.2. Subjects

RBAC can be bound to:

• users;
• groups;
• ServiceAccounts.

Example:

subjects:
  - kind: User
    name: alice@example.com
  - kind: Group
    name: platform-admins
  - kind: ServiceAccount
    name: app
    namespace: demo

9.3. Dangerous Permissions

9.4. RBAC Review

kubectl get role,rolebinding -A
kubectl get clusterrole,clusterrolebinding
kubectl describe clusterrolebinding cluster-admin
kubectl get clusterrolebinding -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.roleRef.name}{"\t"}{range .subjects[*]}{.kind}:{.namespace}:{.name}{","}{end}{"\n"}{end}'

Search for wildcards:

kubectl get clusterrole -o yaml | grep -n "\*"
kubectl get role -A -o yaml | grep -n "\*"

Check Secret access:

kubectl get clusterrole,role -A -o yaml | grep -n -B5 -A10 "secrets"

Check impersonation, bind, and escalate:

kubectl get clusterrole,role -A -o yaml | grep -n -B5 -A10 "impersonate\|bind\|escalate"

9.5. Designing Safe Roles

Process:

1. Describe the use case.
2. Identify required API resources.
3. Use Role instead of ClusterRole when namespace scope is sufficient.
4. Use only specific verbs.
5. Avoid wildcards.
6. Separate reading, writing, deploying, and administration.
7. Test with kubectl auth can-i.
8. Document owner, purpose, and expiry date.

9.6. Example: Developer Role in a Namespace

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer-workload-editor
  namespace: team-a
  labels:
    security.example.com/owner: platform
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

This role does not allow:

• reading Secrets;
• modifying RBAC;
• direct pod exec;
• modifying NetworkPolicies;
• modifying namespace labels;
• modifying cluster objects.

9.7. Break-Glass Model

Break-glass access is emergency access. It must not be daily access.

Minimum rules:

• few people;
• MFA;
• separate accounts;
• time-limited activation;
• audit and alerting;
• mandatory post-documentation;
• regular test exercise;
• immediate revocation when someone leaves or changes roles.

9.8. RBAC Anti-Patterns

10. ServiceAccounts and Workload Identities

ServiceAccounts are identities for processes in Pods. They are not user accounts for humans.

10.1. Risks

• the default ServiceAccount is used everywhere;
• tokens are automatically mounted into Pods;
• ServiceAccounts have overly broad roles;
• tokens appear in logs, debug dumps, or container filesystems;
• cloud access runs through node roles instead of workload identities.

10.2. Standard Rules

10.3. ServiceAccount Without Token Mount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: frontend
  namespace: shop
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: shop
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      serviceAccountName: frontend
      automountServiceAccountToken: false
      containers:
        - name: app
          image: ghcr.io/example/frontend:1.0.0

10.4. ServiceAccount with Minimal API Access

apiVersion: v1
kind: ServiceAccount
metadata:
  name: config-reader
  namespace: shop
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-configmaps
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: config-reader-read-configmaps
  namespace: shop
subjects:
  - kind: ServiceAccount
    name: config-reader
    namespace: shop
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-configmaps

10.5. Cloud Workload Identities

flowchart LR
    Pod[Pod] --> SA[Kubernetes ServiceAccount]
    SA --> OIDC[Cluster OIDC Issuer]
    OIDC --> CloudIAM[Cloud IAM Trust]
    CloudIAM --> Resource[Cloud Resource]

10.6. Review Commands

kubectl get serviceaccounts -A
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.serviceAccountName}{"\t"}{.spec.automountServiceAccountToken}{"\n"}{end}'
kubectl auth can-i --as system:serviceaccount:shop:frontend --list -n shop
kubectl get rolebinding,clusterrolebinding -A -o yaml | grep -n "ServiceAccount"

11. Namespace and Tenant Isolation

Namespaces are not a hard security boundary like separate clusters. They are an administrative and policy boundary.

11.1. What Namespaces Are Suitable For

• team separation;
• RBAC scope;
• ResourceQuota;
• LimitRange;
• NetworkPolicies;
• Pod Security Admission;
• Secret scope;
• GitOps app boundaries;
• cost and ownership labels.

11.2. What Namespaces Do Not Fully Isolate

Namespaces do not automatically separate:

• node kernel;
• container runtime;
• CNI dataplane;
• cluster-wide resources;
• CRDs;
• webhooks;
• storage backends;
• cloud IAM;
• cluster-admin access.

For real tenant isolation, you need additional controls or separate clusters.

11.3. Namespace Baseline

apiVersion: v1
kind: Namespace
metadata:
  name: team-a-prod
  labels:
    owner.example.com/team: team-a
    environment.example.com/stage: prod
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-a-prod
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.memory: 16Gi
    pods: "50"
    persistentvolumeclaims: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a-prod
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 512Mi
      max:
        memory: 2Gi

11.4. Multi-Tenancy Models

11.5. Namespace Review

kubectl get ns --show-labels
kubectl get resourcequota,limitrange -A
kubectl get networkpolicy -A
kubectl get rolebinding -A
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}'

12. Pod Security Standards and Pod Security Admission

Pod Security Standards define three profiles:

Pod Security Admission enforces these standards through namespace labels.

12.1. Modes

12.2. Migrating from Baseline to Restricted

For existing platforms, usually proceed incrementally:

1. set warn=restricted;
2. evaluate warnings;
3. set audit=restricted;
4. document exceptions;
5. adjust workloads;
6. enable enforce=baseline or enforce=restricted;
7. add regression tests in CI.

12.3. Namespace with Warning Instead of Immediate Blocking

apiVersion: v1
kind: Namespace
metadata:
  name: migration-test
  labels:
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest

12.4. Privileged Exceptions

Some platform components need more rights:

• CNI agents;
• CSI node plugins;
• node-exporters with host mounts;
• runtime security agents;
• service mesh CNI;
• kube-proxy or eBPF agents;
• storage or backup agents.

Rules for exceptions:

• dedicated namespace;
• dedicated owner;
• review date;
• rationale;
• minimal app-team write access;
• enforce=privileged only for system namespaces;
• no broad admission exceptions.

12.5. Test a Forbidden Pod

apiVersion: v1
kind: Pod
metadata:
  name: should-fail
  namespace: app-prod
spec:
  containers:
    - name: shell
      image: busybox
      command: ["sleep", "3600"]
      securityContext:
        privileged: true

kubectl apply -f should-fail.yaml

Expected with enforce=restricted: the Pod is rejected.

13. SecurityContext, Linux Isolation, and Container Hardening

Pod Security Admission is only one layer. Workloads should declare secure runtime settings themselves.

13.1. Recommended Container SecurityContext

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  capabilities:
    drop:
      - ALL
  seccompProfile:
    type: RuntimeDefault

seccompProfile can be set at Pod or container level. runAsNonRoot and runAsUser must match the image.

13.2. Pod SecurityContext

spec:
  securityContext:
    runAsNonRoot: true
    fsGroup: 10001
    fsGroupChangePolicy: OnRootMismatch
    seccompProfile:
      type: RuntimeDefault

13.3. Capabilities

Linux capabilities allow selected privileged actions without full root context. In Kubernetes, drop all capabilities by default and add only justified exceptions.

Example justified exception:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

Often better: make the application listen on port 8080 and map Service port 80 to it.

13.4. Read-only Root Filesystem

A read-only root filesystem reduces persistence and manipulation in the container.

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp
  - name: cache
    mountPath: /app/cache
volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

13.5. Avoid Host Namespaces

Avoid in application workloads:

hostNetwork: true
hostPID: true
hostIPC: true

These options increase access and visibility on the node and are typically acceptable only for infrastructure components.

13.6. Avoid hostPath

hostPath connects containers directly to node filesystems. It is dangerous.

Problematic example:

volumes:
  - name: host-root
    hostPath:
      path: /
      type: Directory

Exceptions include DaemonSets for monitoring, CNI, CSI, runtime detection, or node agents. Every exception needs review.

13.7. User Namespaces

User namespaces can separate user IDs inside the container from user IDs on the host. This can make a process appear as root inside the container without being root on the host. Verify support in your Kubernetes version, runtime, distribution, and CNI before using this as a standard.

apiVersion: v1
kind: Pod
metadata:
  name: userns-demo
spec:
  hostUsers: false
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]

13.8. AppArmor, SELinux, and seccomp

Example seccomp:

securityContext:
  seccompProfile:
    type: RuntimeDefault

13.9. Reviewing Workload Hardening

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.securityContext.runAsNonRoot}{"\t"}{.spec.securityContext.seccompProfile.type}{"\n"}{end}'

More detail with jq:

kubectl get pods -A -o json | jq -r '
  .items[] |
  [.metadata.namespace,
   .metadata.name,
   (.spec.containers[]?.securityContext.privileged // false),
   (.spec.containers[]?.securityContext.allowPrivilegeEscalation // "unset"),
   (.spec.containers[]?.securityContext.readOnlyRootFilesystem // "unset")]
  | @tsv'

14. Admission Policy, Policy as Code, and Governance

Admission is the technical place where unsafe resources can be blocked before they start.

14.1. Policy Options

14.2. ValidatingAdmissionPolicy: Disallow latest Tags

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: disallow-latest-image-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "object.spec.containers.all(c, !c.image.endsWith(':latest'))"
      message: "Container images must not use the latest tag. Use a versioned tag or digest."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: disallow-latest-image-tag-binding
spec:
  policyName: disallow-latest-image-tag
  validationActions:
    - Deny
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - kube-system

14.3. Kyverno Example: Enforce runAsNonRoot

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-run-as-non-root
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-run-as-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Pods must set runAsNonRoot=true."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true

14.4. Kyverno Example: Approved Registries

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: only-approved-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from approved registries."
        pattern:
          spec:
            containers:
              - image: "ghcr.io/my-org/* | registry.example.com/*"

Adjust syntax and version to your Kyverno version. Always test policies in audit or warning mode first.

14.5. Gatekeeper Example: ConstraintTemplate

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg}] {
          required := input.parameters.labels[_]
          not input.review.object.metadata.labels[required]
          msg := sprintf("Label %v is missing", [required])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-owner-and-env
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels:
      - owner.example.com/team
      - environment.example.com/stage

14.6. Policy Lifecycle

flowchart LR
    Draft[Policy draft] --> Audit[Audit/warn mode]
    Audit --> Report[Evaluate policy reports]
    Report --> Fix[Adjust workloads]
    Fix --> Enforce[Enable enforce]
    Enforce --> Monitor[Monitor]
    Monitor --> Review[Regular review]
    Review --> Draft

14.7. Governance Rules

• Every policy has an owner, purpose, severity, and exception process.
• Policies are versioned and tested.
• System namespaces are handled separately.
• failurePolicy is chosen deliberately.
• Policies are validated before cluster upgrades.
• Admission controllers have PDBs, monitoring, and resource limits.
• Policy exceptions are time-bound.

15. Secrets, Keys, and Confidential Data

Kubernetes Secrets are intended for confidential data, but they are not automatically sufficiently protected. Base64 is encoding, not encryption.

15.1. Secret Risks

15.2. Secret Usage Patterns

15.3. Kubernetes Secret with Demo Value

kubectl create secret generic demo-secret \
  -n app-prod \
  --from-literal=username=demo \
  --from-literal=password='not-for-production' \
  --dry-run=client -o yaml

15.4. Do Not Print Secrets

Avoid in training:

kubectl get secret app-secret -o yaml
kubectl get secret app-secret -o jsonpath='{.data.password}' | base64 -d

If you must demonstrate it, use dummy values deliberately and clear shell history in lab environments.

15.5. Encryption at Rest

Example encryption configuration for self-managed clusters:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

API Server integration:

- --encryption-provider-config=/etc/kubernetes/encryption/encryption-config.yaml

After enabling encryption, rewrite existing objects so they are stored encrypted:

kubectl get secrets -A -o json | kubectl replace -f -

In production, prefer KMS providers or provider-specific key-management integrations where available.

15.6. External Secret Systems

15.7. Secret Rotation

Rotation includes:

1. create a new secret;
2. make the application compatible;
3. roll out the secret;
4. reload or restart the workload;
5. invalidate the old secret;
6. check access and logs;
7. update documentation.

Example restart after a Secret change:

kubectl rollout restart deployment/app -n app-prod
kubectl rollout status deployment/app -n app-prod

15.8. Short Secret Leak Runbook

1. Treat the secret as compromised.
2. Disable or rotate the password/API key.
3. Review logs and audit.
4. Redeploy affected workloads.
5. Clean Git history and artifacts if the secret appeared there.
6. Fix the root cause.
7. Improve detection.

16. etcd and Control Plane Protection

etcd stores cluster state. Anyone who can read or manipulate etcd effectively controls the cluster.

16.1. etcd Risks

• plaintext Secrets when encryption at rest is missing;
• unprotected snapshots;
• access to etcd certificates;
• missing TLS protection;
• no restore test;
• insecure control-plane nodes;
• missing monitoring for leader, latency, disk, and quorum.

16.2. Self-Managed Clusters

Check:

sudo ls -l /etc/kubernetes/pki/etcd
sudo crictl ps | grep etcd
sudo grep -n "etcd" /etc/kubernetes/manifests/etcd.yaml

etcd snapshot:

sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd/etcd-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Snapshot status:

ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd/<snapshot>.db -w table

Security rules:

• encrypt snapshots;
• do not keep snapshots only on control-plane nodes;
• test restores regularly;
• restrict access to etcd certificates;
• do not run normal workloads on control-plane nodes;
• do not expose API Server or etcd publicly.

16.3. Managed Kubernetes

For EKS, AKS, GKE, and similar services, the provider manages the control plane. You still control:

• API endpoint reachability;
• cloud IAM;
• audit/control-plane logs;
• Secrets encryption/KMS options;
• add-ons;
• nodes;
• workload identities;
• NetworkPolicies;
• RBAC;
• backups of Kubernetes resources and PV data.

16.4. Control Plane Hardening in kubeadm

sudo ls -l /etc/kubernetes/manifests
sudo grep -n "authorization-mode\|admission\|audit\|encryption" /etc/kubernetes/manifests/kube-apiserver.yaml
sudo grep -n "profiling\|bind-address" /etc/kubernetes/manifests/kube-controller-manager.yaml
sudo grep -n "profiling\|bind-address" /etc/kubernetes/manifests/kube-scheduler.yaml

Typical goals:

• RBAC enabled;
• NodeAuthorizer enabled;
• suitable admission plugins;
• audit enabled;
• profiling disabled or isolated;
• bind-address not unnecessarily open;
• certificates checked regularly.

17. Node, Host, and Runtime Hardening

A node is a critical security boundary. If an attacker gets node access, all Pods on that node are at risk.

17.1. Node Baseline

17.2. kubelet Protection

kubelet exposes an HTTPS API. This API can disclose sensitive information and may allow powerful actions depending on permissions.

Check:

sudo cat /var/lib/kubelet/config.yaml | grep -E "anonymous|authorization|authentication" -n
sudo ss -lntp | grep kubelet

Target values:

authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook

Review RBAC for nodes/proxy strictly:

kubectl get clusterrole -o yaml | grep -n -B5 -A15 "nodes/proxy"
kubectl get clusterrolebinding -o yaml | grep -n -B5 -A15 "nodes"

17.3. Container Runtime

Check runtime:

kubectl get nodes -o wide
crictl info
crictl ps
systemctl status containerd --no-pager
systemctl status crio --no-pager

Recommendations:

• no Docker socket mounts in Pods;
• patch runtime regularly;
• no insecure registries unless explicitly isolated;
• registry mirrors with TLS;
• cgroup driver consistent with kubelet;
• centralize runtime logs.

17.4. Docker Socket Anti-Pattern

Very critical:

volumeMounts:
  - name: docker-sock
    mountPath: /var/run/docker.sock
volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock

Anyone who controls the Docker or container runtime socket can often compromise the host. Use instead:

• rootless build tools;
• BuildKit in an isolated environment;
• Kaniko with restrictions;
• cloud build services;
• dedicated build clusters;
• separate runners without production access.

17.5. Node Isolation

Separate nodes by risk:

Example taint:

kubectl taint nodes node-a dedicated=system:NoSchedule

Example toleration:

tolerations:
  - key: dedicated
    operator: Equal
    value: system
    effect: NoSchedule

18. Network Security

Without NetworkPolicies, pod communication is open in many clusters. NetworkPolicies work only if the CNI plugin enforces them.

18.1. Network Zones

flowchart TB
    Internet[Internet] --> Edge[Load Balancer / WAF]
    Edge --> Ingress[Ingress Controller / Gateway]
    Ingress --> App[App Namespace]
    App --> DB[Database Namespace]
    App --> DNS[CoreDNS]
    App --> Egress[Egress Gateway / NAT]
    Egress --> External[External APIs]
    Monitor[Monitoring] --> App
    Security[Runtime Security] --> App

18.2. NetworkPolicy Basics

NetworkPolicies select Pods by labels. Once a policy applies to a Pod for ingress or egress, that Pod is isolated in that direction and only explicitly allowed traffic works.

18.3. Default Deny per Namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress-egress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

18.4. Allow Ingress from Ingress Controller

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller-to-web
  namespace: team-a
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

18.5. Allow App to Database

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-postgres
  namespace: database
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              owner.example.com/team: team-a
          podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 5432

18.6. Egress to External API

The standard Kubernetes NetworkPolicy model can match IP blocks, but not FQDNs. FQDN-based egress control requires CNI extensions, an egress gateway, proxy, or service mesh.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-payment-provider
  namespace: team-a
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 203.0.113.0/24
      ports:
        - protocol: TCP
          port: 443

18.7. CNI-Specific Extensions

18.8. Egress Strategies

18.9. Network Troubleshooting

kubectl get networkpolicy -A
kubectl describe networkpolicy -n team-a
kubectl run netshoot -n team-a --rm -it --image=nicolaka/netshoot --restart=Never -- bash
nslookup kubernetes.default.svc.cluster.local
curl -v http://service.namespace.svc.cluster.local:8080

Always check:

• Pod labels;
• namespace labels;
• CNI support;
• DNS egress;
• ingress-controller namespace;
• Service port versus targetPort;
• node firewalls;
• cloud security groups.

19. Ingress, Gateway, TLS, and External Exposure

Ingress and Gateway API are central attack surfaces because they expose internal workloads externally.

19.1. Ingress Risks

• TLS missing or misconfigured;
• wildcard hosts route unexpectedly;
• admin endpoints are public;
• ingress controller has overly broad RBAC;
• unsafe annotations allow snippet injection;
• rate limits and body limits are missing;
• authentication is missing for internal tools.

19.2. Gateway API Role Model

flowchart LR
    Infra[Infrastructure team] --> GatewayClass[GatewayClass]
    Platform[Platform team] --> Gateway[Gateway]
    AppTeam[App team] --> HTTPRoute[HTTPRoute]
    GatewayClass --> Gateway
    Gateway --> HTTPRoute
    HTTPRoute --> Service[Service]

19.3. TLS Standard

• redirect HTTP to HTTPS;
• issue and renew TLS certificates automatically;
• wildcard certificates only after review;
• do not route private services publicly;
• enable HSTS only deliberately;
• alert on certificate expiry.

19.4. cert-manager Example

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: app-example-com
  namespace: app-prod
spec:
  secretName: app-example-com-tls
  dnsNames:
    - app.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

19.5. Gateway API Example

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public-gateway
  namespace: gateways
spec:
  gatewayClassName: nginx
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app
  namespace: app-prod
spec:
  parentRefs:
    - name: public-gateway
      namespace: gateways
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: app
          port: 8080

19.6. External Endpoints in Local Environments

20. Image, Registry, and Supply Chain Security

Kubernetes can only run what you give it. Unsafe images are one of the most common causes of cluster compromise.

20.1. Supply Chain Flow

flowchart LR
    Dev[Developer] --> Repo[Git Repository]
    Repo --> CI[CI Runner]
    CI --> Build[Image Build]
    Build --> Scan[Vulnerability Scan]
    Build --> SBOM[SBOM]
    Build --> Sign[Signature]
    Sign --> Registry[Registry]
    Registry --> Admission[Admission Verification]
    Admission --> Cluster[Kubernetes Cluster]
    Cluster --> Runtime[Runtime Detection]

20.2. Image Baseline

20.3. Use Image with Digest

containers:
  - name: app
    image: registry.example.com/team/app@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

20.4. Trivy Scan

trivy image registry.example.com/team/app:1.2.3
trivy image --severity HIGH,CRITICAL --exit-code 1 registry.example.com/team/app:1.2.3
trivy fs .
trivy config ./k8s

20.5. SBOM with Syft

syft registry.example.com/team/app:1.2.3 -o spdx-json > sbom.spdx.json
syft registry.example.com/team/app:1.2.3 -o cyclonedx-json > sbom.cyclonedx.json

20.6. Sign and Verify with Cosign

cosign sign registry.example.com/team/app:1.2.3
cosign verify registry.example.com/team/app:1.2.3
cosign verify-attestation registry.example.com/team/app:1.2.3

20.7. Admission for Signed Images

Possible approaches:

• Kyverno verifyImages;
• Sigstore policy-controller;
• Connaisseur;
• cloud-native Binary Authorization;
• custom Validating Webhooks.

20.8. Registry RBAC

20.9. Unsafe Image Patterns

• latest;
• unversioned base images;
• root user in image;
• package manager and shell in runtime image without need;
• SSH server in container;
• debug tools in production;
• secrets in layers;
• non-reproducible builds;
• pull from public registries without mirror and verification.

21. CI/CD and GitOps Security

CI/CD and GitOps are part of the Kubernetes attack surface. A compromised deployment controller can change production clusters.

21.1. CI/CD Risks

• runner has access to production secrets;
• pull requests can run untrusted code with secrets;
• deployment tokens are static and long-lived;
• build images are not hardened;
• artifacts are not signed;
• branch protection is missing.

21.2. CI/CD Baseline

21.3. GitOps Risks

21.4. Argo CD Baseline

• enable SSO;
• disable or strongly protect local admin accounts;
• use AppProjects;
• restrict target namespaces;
• restrict cluster-scoped resources;
• grant minimal repository access;
• review sync windows for critical environments;
• integrate notifications and audit.

21.5. Flux Baseline

• limit Git access with deploy keys;
• use SOPS or External Secrets;
• review Kustomization scope;
• minimize controller RBAC;
• use image automation only with a review process;
• monitor drift and reconciliation events.

22. Storage and Data Security

Storage is not only availability. It affects data classification, encryption, access, backup, and deletion.

22.1. Storage Risks

• PV contains personal or secret data;
• StorageClass provisions unencrypted volumes;
• ReclaimPolicy deletes data unexpectedly or retains it too long;
• backup contains secrets or customer data;
• snapshot permissions are too broad;
• Pods mount PVCs they should not access;
• hostPath bypasses storage controls.

22.2. StorageClass Security

kubectl get storageclass -o yaml
kubectl get pv,pvc -A
kubectl get volumesnapshotclass

Check:

22.3. Minimize PVC Access

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: app-prod
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: encrypted-standard

22.4. Limit Snapshot RBAC

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: snapshot-reader
  namespace: app-prod
rules:
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch"]

22.5. Data Classification

23. Observability, Audit, Detection, and Forensics

Security without visibility is not reliable. You need metrics, logs, audit events, and runtime signals.

23.1. Signal Types

23.2. Audit Policy Example

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "admissionregistration.k8s.io"
        resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
  - level: Metadata
    omitStages:
      - RequestReceived

Do not unnecessarily log Secret values. Audit level RequestResponse can contain sensitive content.

23.3. Alert Categories

23.4. PrometheusRule Examples

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-security-alerts
  namespace: monitoring
spec:
  groups:
    - name: kubernetes-security
      rules:
        - alert: KubeAPIServerAuditErrorsHigh
          expr: increase(apiserver_audit_event_total{stage="Panic"}[5m]) > 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "API Server audit panic events"
            description: "The API Server has produced audit events in panic stage."
        - alert: TooManyPodsPending
          expr: count(kube_pod_status_phase{phase="Pending"}) > 20
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Many Pods are Pending"
            description: "This can indicate quota, scheduling, CNI, or policy issues."

23.5. Forensic Principles

• Change as little state as possible.
• Preserve logs, events, manifests, and image digests.
• Record time ranges and time zones.
• Rotate credentials only after preserving necessary evidence, if risk allows.
• Isolate and replace compromised nodes.
• Use snapshots only after privacy and approval processes.

23.6. Evidence Collection for a Suspicious Pod

NS=app-prod
POD=demo-app-abc123
mkdir -p incident-$NS-$POD
kubectl get pod "$POD" -n "$NS" -o yaml > incident-$NS-$POD/pod.yaml
kubectl describe pod "$POD" -n "$NS" > incident-$NS-$POD/describe.txt
kubectl logs "$POD" -n "$NS" --all-containers > incident-$NS-$POD/logs-current.txt
kubectl logs "$POD" -n "$NS" --all-containers --previous > incident-$NS-$POD/logs-previous.txt || true
kubectl get events -n "$NS" --sort-by=.lastTimestamp > incident-$NS-$POD/events.txt

24. Runtime Security

Runtime security detects suspicious behavior after a workload has started.

24.1. What Runtime Security Detects

• shell in a container;
• new executable files;
• access to sensitive paths;
• unusual network connections;
• process starts outside the expected profile;
• privilege escalation attempts;
• container escape indicators;
• Kubernetes API misuse;
• suspicious cloud API calls.

24.2. Falco Baseline

Falco observes kernel events and other sources and generates alerts using rules.

Example rule:

- rule: Shell spawned in container
  desc: A shell was started in a container
  condition: container.id != host and proc.name in (bash, sh, zsh, csh)
  output: Shell started in container (user=%user.name command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, shell, mitre_execution]

24.3. Tetragon and eBPF

Tetragon is especially relevant in Cilium environments. It can collect process, network, and security events using eBPF and add Kubernetes context.

Typical use cases:

• process execution in Pods;
• network connections;
• file or syscall-adjacent detection;
• policy-adjacent runtime enforcement depending on setup.

24.4. Runtime Detection Architecture

flowchart LR
    Kernel[Linux Kernel Events] --> Agent[Runtime Agent: Falco/Tetragon]
    K8s[Kubernetes API Context] --> Agent
    Runtime[Container Runtime] --> Agent
    Agent --> Alerts[Alerts]
    Alerts --> SIEM[SIEM / Log Backend]
    Alerts --> IR[Incident Response]

24.5. Runtime Alert Triage

25. Backup, Restore, and Disaster Recovery Security

Backups are security-critical. They often contain the most complete data set.

25.1. What Must Be Backed Up

25.2. Backup Security Rules

• Encrypt backups.
• Strongly restrict backup access.
• Test restores regularly.
• Define backup retention.
• Use offsite copies or a separate region.
• Verify backup integrity.
• Enable ransomware protection for backup buckets where available.
• Treat Secrets and KMS keys separately.

25.3. Velero Backup Example

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-prod
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - app-prod
    ttl: 720h0m0s
    storageLocation: default
    snapshotVolumes: true

25.4. Restore Test

velero backup create app-prod-manual --include-namespaces app-prod
velero backup describe app-prod-manual
velero restore create app-prod-restore-test --from-backup app-prod-manual --namespace-mappings app-prod:app-prod-restore
velero restore describe app-prod-restore-test
kubectl get all,pvc,secret,configmap -n app-prod-restore

25.5. DR Security Questions

• Who may trigger restore?
• Could a restore reintroduce old vulnerable images?
• Is secret rotation required after restore?
• Is RBAC restored?
• Are old admission policies restored?
• Is there an isolated restore test cluster?
• Are backups available after a region/provider outage?

26. Patch, Update, and Vulnerability Management

Security is a lifecycle topic. Kubernetes versions, node images, add-ons, and application images must be maintained.

26.1. Patch Layers

26.2. Vulnerability Workflow

flowchart LR
    Detect[CVE or advisory] --> Assess[Check exposure]
    Assess --> Risk[Assess severity and exposure]
    Risk --> Plan[Patch plan]
    Plan --> Test[Test cluster / staging]
    Test --> Rollout[Production rollout]
    Rollout --> Verify[Validation]
    Verify --> Document[Documentation]

26.3. Kubernetes Upgrade Security

Before upgrade:

kubectl version
kubectl get nodes -o wide
kubectl get apiservices
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get crd
kubectl get events -A --sort-by=.lastTimestamp | tail -100

Check:

• version skew;
• deprecated APIs;
• admission compatibility;
• CNI/CSI compatibility;
• backup and restore;
• node drain behavior;
• PodDisruptionBudgets;
• Cluster Autoscaler compatibility;
• security-relevant release notes.

26.4. Image CVE Triage

Not every CVE is equally critical. Evaluate:

• Is the vulnerable library actually used?
• Is the code path reachable?
• Is the Pod publicly reachable?
• Does the container run as root?
• Are NetworkPolicy and egress controls in place?
• Is runtime detection active?
• Is there a fix or only mitigation?
• Can you rebuild with a newer base image?

26.5. Example Patch SLAs

27. Security by Kubernetes Variant

27.1. kubeadm

kubeadm gives you a lot of control and a lot of responsibility.

Important controls:

• OS hardening;
• secure containerd/CRI-O configuration;
• kubelet webhook authn/authz;
• API audit;
• Secrets encryption at rest;
• etcd snapshots;
• CNI with NetworkPolicy;
• secure control-plane endpoint;
• certificate rotation;
• upgrade discipline.

Review commands:

kubeadm version
kubeadm certs check-expiration
kubectl get nodes -o wide
sudo ls /etc/kubernetes/manifests
sudo journalctl -u kubelet --no-pager | tail -100

27.2. k3s

k3s is lightweight and often suitable for edge, labs, and smaller clusters.

Security focus:

• protect server token;
• secure embedded etcd or external DB;
• consciously enable or disable Traefik/ServiceLB;
• configure local registry mirrors securely;
• minimize node access;
• check Secret encryption;
• review systemd unit and install scripts.

sudo systemctl status k3s
sudo cat /etc/rancher/k3s/config.yaml
kubectl get nodes -o wide
kubectl get pods -A

27.3. RKE2

RKE2 is more oriented toward production-like and hardened environments.

Security focus:

• review CIS profile;
• use SELinux where appropriate;
• version static Pod manifests and add-ons;
• protect server and agent tokens;
• encrypt and store etcd snapshots safely;
• separate system namespaces.

27.4. k0s

k0s is a lean Kubernetes distribution with its own cluster configuration.

Security focus:

• separate controller and worker roles;
• version k0s configuration;
• handle join tokens as short-lived;
• choose CNI and storage deliberately;
• check kubelet and runtime;
• test backup/restore of cluster state.

27.5. Talos Linux

Talos is an immutable OS for Kubernetes. Management is through the Talos API, not SSH.

Security focus:

• treat MachineConfig as sensitive desired state;
• protect talosconfig;
• no classic SSH management;
• restrict API access;
• evaluate disk encryption;
• store cluster and machine secrets securely;
• run upgrades declaratively and reproducibly.

27.6. Docker Desktop Kubernetes

Suitable for local development, not production security exercises with real secrets.

Security focus:

• do not use production kubeconfigs or tokens in Docker Desktop;
• clean up local images and volumes regularly;
• name Kubernetes contexts clearly;
• do not expose demo ingresses with real data.

27.7. MicroK8s

MicroK8s is practical for local labs, edge, and Ubuntu-oriented environments.

Security focus:

• enable add-ons deliberately;
• review local user groups;
• control snap updates;
• secure registry and ingress;
• check host firewall.

28. Security by Environment

28.1. WSL on Windows 11

WSL is a strong learning and CLI environment, but not a complete production substitute.

Risks:

• Windows host and WSL share attack surfaces;
• files under /mnt/c have different permissions and performance characteristics;
• Docker Desktop integration extends the trust boundary;
• Windows hosts file can affect local name resolution;
• real cloud credentials can easily sit in the home directory.

Recommendations:

• no real production secrets in WSL labs;
• name kubeconfig contexts clearly;
• do not bypass Windows Defender/EDR;
• clearly mark local demo domains, for example *.127.0.0.1.nip.io or *.localtest.me;
• modify the Windows hosts file only deliberately and with documentation;
• do not fill Git repositories with real secrets.

Back up the Windows hosts file from WSL:

sudo cp /mnt/c/Windows/System32/drivers/etc/hosts /mnt/c/Windows/System32/drivers/etc/hosts.k8s-backup

Add entry if you have elevated rights:

printf "127.0.0.1 app.local\n" | sudo tee -a /mnt/c/Windows/System32/drivers/etc/hosts

Rollback:

sudo cp /mnt/c/Windows/System32/drivers/etc/hosts.k8s-backup /mnt/c/Windows/System32/drivers/etc/hosts

28.2. kind

kind runs Kubernetes nodes as containers. It is ideal for policy, admission, CI, and manifest tests.

Security notes:

• the host Docker/Podman environment is the real security boundary;
• no real secrets;
• set extraPortMappings deliberately;
• use local registry only with demo images;
• delete clusters after tests.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        protocol: TCP
      - containerPort: 443
        hostPort: 8443
        protocol: TCP

28.3. minikube

minikube is suitable for local development and labs.

Security notes:

• driver determines risk: Docker, VirtualBox, VMware, Hyper-V, SSH;
• minikube tunnel may need elevated rights;
• add-ons add components with RBAC;
• clean up local images and secrets regularly.

minikube start --driver=docker
minikube addons enable ingress
minikube addons enable metrics-server
minikube status

28.4. k3d

k3d runs k3s in Docker containers.

Security notes:

• protect Docker host;
• set port mappings deliberately;
• choose Traefik or your own ingress solution deliberately;
• delete clusters after tests.

k3d cluster create lab \
  --servers 1 \
  --agents 2 \
  -p "8080:80@loadbalancer" \
  -p "8443:443@loadbalancer"

28.5. Local VM

Security focus:

• VM snapshots before risky labs;
• host-only network for private labs;
• NAT/bridged only when needed;
• SSH key instead of password;
• firewall active;
• no private cloud credentials in snapshots.

28.6. Remote VM

Security focus:

• cloud firewall first, host firewall additionally;
• SSH only from known IPs;
• do not open API Server publicly;
• do not expose Kubernetes Dashboard publicly;
• protect local kubeconfigs;
• encrypt disks;
• restrict snapshot access.

28.7. Bare Metal

Security focus:

• secure firmware/BIOS/UEFI and BMC;
• isolate management network;
• protect PXE/provisioning network;
• evaluate disk encryption;
• consider physical access;
• evaluate hardware supply chain;
• audit out-of-band access.

29. Cloud-Specific Security

Managed Kubernetes reduces control-plane operations but not your platform responsibility.

29.1. AWS EKS

Important controls:

• private or restricted API endpoints;
• model IAM Identity Center or IAM-based authentication cleanly;
• review EKS Access Entries instead of relying on historically grown aws-auth disorder;
• use EKS Pod Identity or IRSA for workloads;
• keep node IAM role minimal;
• review Security Groups and NACLs;
• enable control plane logs;
• use KMS for Secrets encryption;
• evaluate GuardDuty EKS Protection;
• ECR scanning and image signatures;
• evaluate managed node groups or Fargate by risk.

Review commands:

aws eks describe-cluster --name <cluster>
aws eks list-access-entries --cluster-name <cluster>
aws eks list-pod-identity-associations --cluster-name <cluster>
aws eks update-kubeconfig --name <cluster> --region <region>
kubectl get nodes -o wide

29.2. Azure AKS

Important controls:

• Microsoft Entra ID integration;
• deliberately choose Azure RBAC for Kubernetes Authorization;
• private cluster or API authorized IP ranges;
• AKS Workload Identity;
• Azure Policy for Kubernetes;
• Microsoft Defender for Containers;
• Azure Key Vault CSI Driver or External Secrets;
• NetworkPolicies with Azure CNI/Cilium/Calico depending on setup;
• node OS upgrades and maintenance windows;
• Container Registry scans and RBAC.

Review commands:

az aks show -g <resource-group> -n <cluster>
az aks get-credentials -g <resource-group> -n <cluster>
az aks update -g <resource-group> -n <cluster> --enable-oidc-issuer --enable-workload-identity
kubectl get nodes -o wide

29.3. Google GKE

Important controls:

• separate IAM and Kubernetes RBAC clearly;
• use Workload Identity Federation for GKE;
• evaluate Autopilot for stronger managed defaults;
• private clusters and Master Authorized Networks;
• Shielded GKE Nodes;
• Binary Authorization;
• Artifact Registry scans;
• Cloud KMS for Secrets and storage;
• Cloud Audit Logs and Security Command Center;
• GKE release channels and maintenance windows.

Review commands:

gcloud container clusters describe <cluster> --region <region>
gcloud container clusters get-credentials <cluster> --region <region>
kubectl get nodes -o wide

29.4. Oracle OKE

Important controls:

• review private API endpoints;
• keep OCI IAM policies minimal;
• worker nodes in private subnets;
• Network Security Groups;
• Vault/KMS integration;
• enable logging and audit;
• check image scanning in OCIR.

29.5. DigitalOcean DOKS

Important controls:

• restrict kubeconfig access;
• review API access and team permissions;
• Container Registry RBAC;
• Load Balancer and firewall rules;
• manage secrets externally;
• verify NetworkPolicies with supported CNI.

29.6. Linode/Akamai LKE

Important controls:

• Cloud Firewall;
• node pool access;
• API and kubeconfig protection;
• Longhorn/CSI storage security;
• registry and secret management;
• add provider monitoring and audit options.

29.7. Multi-Cloud

Multi-cloud does not automatically improve security. It increases complexity.

Required standards:

• consistent RBAC role models;
• consistent labels and namespaces;
• central policy library;
• secret backend per provider or federated;
• centralized audit-log collection;
• identical admission gates;
• clear cluster inventory;
• provider-specific exceptions documented.

30. OpenShift, OKD, and SCC

OpenShift adds its own platform concepts to Kubernetes. Security Context Constraints, Routes, Operators, Projects, and OAuth are especially relevant.

30.1. SCC Instead of Only Pod Security Admission

Security Context Constraints control under which security conditions Pods are allowed to run. They are OpenShift-specific and historically central to Pod security.

Review:

oc get scc
oc describe scc restricted-v2
oc adm policy who-can use scc restricted-v2

30.2. OpenShift Security Baseline

30.3. OpenShift Anti-Patterns

• broadly granting anyuid;
• privileged SCC for app namespaces;
• ClusterRoleBindings without a group model;
• operators without lifecycle plan;
• routes without TLS;
• build secrets in the wrong namespace;
• local admin accounts without audit.

31. Compliance, Benchmarks, and Audits

Compliance is not a substitute for security engineering. It helps make controls measurable.

31.1. CIS Kubernetes Benchmark

The CIS Kubernetes Benchmark contains concrete configuration recommendations for Kubernetes and provider variants.

Typical areas:

• API Server;
• Controller Manager;
• Scheduler;
• etcd;
• kubelet;
• policies;
• RBAC;
• NetworkPolicies;
• Secrets;
• Pod Security.

31.2. kube-bench

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl get pods -n default
kubectl logs job/kube-bench

For production, review manifests before applying, pin image versions, and control scope.

31.3. NSA/CISA-Hardening-Oriented Controls

Typical topics:

• scan containers and Pods for vulnerabilities and misconfiguration;
• least privilege for containers and Pods;
• network segmentation;
• strong authentication;
• audit logging;
• protect Secrets;
• avoid dangerous workload options;
• harden nodes.

31.4. Audit Artifacts

31.5. Audit Export

mkdir -p audit-export
kubectl get ns --show-labels > audit-export/namespaces.txt
kubectl get role,rolebinding -A -o yaml > audit-export/rbac-namespaced.yaml
kubectl get clusterrole,clusterrolebinding -o yaml > audit-export/rbac-cluster.yaml
kubectl get networkpolicy -A -o yaml > audit-export/networkpolicies.yaml
kubectl get pods -A -o yaml > audit-export/pods.yaml
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o yaml > audit-export/webhooks.yaml
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding -o yaml > audit-export/validating-admission-policies.yaml || true
kubectl get resourcequota,limitrange -A -o yaml > audit-export/quotas-limits.yaml

Protect these exports. They can contain Secrets, internal names, images, and architecture details.

32. Tooling Matrix

32.1. Kubescape Example

kubescape scan framework nsa
kubescape scan framework cis
kubescape scan --submit=false

32.2. Polaris Example

polaris audit --audit-path ./manifests --format=pretty

32.3. Checkov Example

checkov -d ./manifests --framework kubernetes
checkov -d ./terraform --framework terraform

32.4. Conftest Example

conftest test deployment.yaml

33. Standard Runbooks

33.1. Runbook: Compromised Pod

Goal: limit damage, preserve evidence, fix root cause.

1. Validate the alert.
2. Capture namespace, Pod, node, image, ServiceAccount.
3. Preserve logs, events, and manifest.
4. Restrict network access.
5. Do not blindly delete the Pod if forensics are required.
6. Review ServiceAccount permissions.
7. Rotate Secrets if access was possible.
8. Check image digest and registry history.
9. Roll out workload with fixed image and policy.
10. Improve admission and runtime rules.

Commands:

NS=<namespace>
POD=<pod>
kubectl get pod "$POD" -n "$NS" -o wide
kubectl describe pod "$POD" -n "$NS"
kubectl logs "$POD" -n "$NS" --all-containers --previous
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.serviceAccountName}{"\n"}'
kubectl auth can-i --as system:serviceaccount:$NS:$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.serviceAccountName}') --list -n "$NS"