Kubernetes Workshop: Advanced + Security – Exercises with Solutions
Reference: 04-advanced-security-EN.md
Difficulty level: Advanced + Security
Format: security review, hands-on lab, walkthrough, and sample solutions
Goal of this exercise document
This exercise document complements the Advanced and Security workshop with practical tasks, review checklists, sample solutions, and walkthroughs. The focus is on threat modeling, API Server and control-plane hardening, RBAC, identity, Pod Security Standards, workload hardening, RuntimeClass, network security, supply-chain security, policy-as-code, Admission Control, Secrets security, runtime detection, compliance, evidence, and incident response.
Work only in a lab cluster. Do not use production Secrets, real customer data, or production cluster contexts.
Prerequisites
You need:
- a lab cluster
kubectl- rights to create Namespaces, ServiceAccounts, RBAC, NetworkPolicies, and test workloads
- optional Kyverno, OPA Gatekeeper, or ValidatingAdmissionPolicy support
- optional Trivy, Grype, Cosign, or a prepared supply-chain demo flow
- optional Falco, Tetragon, Tracee, or comparable runtime signals
Recommended Namespace:
export NS=workshop-security
kubectl create namespace "$NS"
kubectl config set-context --current --namespace="$NS"
Exercise overview
| No. | Topic | Result |
|---|---|---|
| 1 | Threat Model | risks by API, Node, workload, network, Secrets, supply chain |
| 2 | RBAC audit | effective rights and escalation paths evaluated |
| 3 | ServiceAccount Token | token automount reduced |
| 4 | Pod Security Restricted | insecure Pods blocked and hardened |
| 5 | Workload hardening | SecurityContext set correctly |
| 6 | NetworkPolicy | Default Deny, DNS, and app traffic controlled |
| 7 | Supply Chain | image, tag, digest, SBOM, scan, and signature evaluated |
| 8 | Admission Policy | required Labels or SecurityContext enforced |
| 9 | Secrets Security | Secret flows and rotation evaluated |
| 10 | Runtime Detection | signals and reactions defined |
| 11 | Compliance Evidence | finding, risk, exception, and fix documented |
| 12 | Incident Response | runbook for compromised Pod and Secret leak |
Part A – Exercises
Exercise 1: Write a Kubernetes Threat Model
Goal
You model security risks separately by Kubernetes attack surfaces.
Scenario
A sample app consists of:
- Namespace
shop-prod, - Deployment
checkout-api, - ServiceAccount
checkout-sa, - Secret
payment-api-token, - Ingress/Gateway route,
- private registry,
- GitOps Controller,
- CI pipeline.
Task
Create threat-model.md with:
- assets and protection value,
- trust boundaries,
- attack surfaces,
- abuse scenarios,
- existing controls,
- missing controls,
- stop criteria,
- evidence.
Exercise 2: Perform RBAC audit
Goal
You identify overly broad rights and privilege-escalation paths.
Task
- Create two ServiceAccounts:
app-saandoverprivileged-sa. - Give
app-saminimal read rights for ConfigMaps. - Intentionally give
overprivileged-sarights to Secrets and RoleBindings. - Check effective rights with
kubectl auth can-i. - Document risks and fixes.
Exercise 3: Reduce ServiceAccount token automount
Goal
You make sure that Pods without Kubernetes API needs do not automatically receive a mounted ServiceAccount token.
Task
- Create a Deployment
no-api-app. - Set
automountServiceAccountToken: false. - Check in the Pod whether
/var/run/secrets/kubernetes.io/serviceaccountexists. - Explain when a Pod needs a token and when it does not.
Exercise 4: Enforce Pod Security Restricted
Goal
You enforce Pod Security Standards in the Namespace and correct an insecure Pod.
Task
- Set Namespace Labels for
restrictedinenforce,audit, andwarn. - Create an insecure Pod with
privileged: true. - Observe the rejection.
- Create a hardened Pod that satisfies Restricted requirements.
- Document the differences.
Exercise 5: Workload hardening with SecurityContext
Goal
You translate workload hardening into concrete Pod and container SecurityContext fields.
Task
Hardening criteria:
- do not run as root,
- forbid privilege escalation,
- read-only root filesystem,
- drop all Linux capabilities,
- seccomp
RuntimeDefault, - temporary writable directory through
emptyDir, - no host namespace usage.
Create a Deployment that meets these criteria.
Exercise 6: NetworkPolicy Default Deny with DNS and app allow
Goal
You reduce lateral movement and allow only required traffic.
Task
- Create
frontendandbackend. - Create Default Deny for Ingress and Egress.
- Allow DNS Egress.
- Allow
frontendtobackendon port 8080. - Block all other Egress.
- Document which additional exceptions would be needed in production.
Exercise 7: Check supply-chain security
Goal
You evaluate a Kubernetes workload from a supply-chain perspective.
Task
For a sample image, document:
- registry,
- repository,
- tag,
- digest,
- pull policy,
- ImagePullSecret,
- scan result,
- SBOM status,
- signature status,
- Admission requirement,
- stop criterion.
Optional with tools:
trivy image <image>
cosign verify <image>@<digest>
Exercise 8: Write an Admission Policy
Goal
You formulate a Policy that blocks insecure or incomplete resources.
Task
Write a Policy for one of these goals:
- required Labels
owner,environment,cost-center, - ban
privileged: true, - require
runAsNonRoot: true, - require resource requests and limits,
- only allowed registries.
Depending on your lab, use Kyverno, OPA Gatekeeper, or a native ValidatingAdmissionPolicy.
Exercise 9: Evaluate Secrets security and rotation
Goal
You distinguish Kubernetes Secret, External Secrets, CSI Secret Store, Sealed Secrets, and SOPS based on a concrete risk.
Task
Create secrets-review.md with:
- Secret inventory,
- who has access,
- where Secrets are mounted,
- whether Secrets could become visible in logs or Events,
- rotation path,
- etcd/backup risk,
- recommendation for a Secret system,
- incident action in case of leak.
Exercise 10: Classify runtime detection
Goal
You define signals that runtime detection or audit logging should detect.
Task
Define detection rules or expected signals for:
- unexpected shell in the container,
- write access to
/etc,/proc, or/sys, - unexpected Egress,
- access to the Kubernetes API with ServiceAccount,
- suspicious
kubectl exec, - OOMKill in a security-relevant Pod.
For each signal, document: risk, data source, first reaction, and evidence.
Exercise 11: Create compliance evidence
Goal
You create a defensible compliance finding with risk, owner, exception, or fix.
Task
Create compliance-evidence.md with at least five findings:
- Namespace without Pod Security Label,
- ServiceAccount with overly broad rights,
- Pod without requests/limits,
- Secret without rotation note,
- workload without NetworkPolicy.
For each finding, document:
- timestamp,
- resource,
- risk,
- evidence,
- owner,
- fix,
- exception with expiration date or remediation deadline.
Exercise 12: Write incident runbook
Goal
You formulate a Kubernetes incident-response path for a compromised Pod and a Secret leak.
Task
Create incident-runbook.md with:
- triage,
- isolation,
- evidence preservation,
- containment,
- Secret and token rotation,
- image and CI check,
- rebuild and redeploy,
- communication,
- lessons learned,
- policy hardening.
Part B – Solutions and walkthroughs
Solution 1: Threat Model
Sample structure:
# Threat Model: checkout-api
## Assets
| Asset | Protection value |
| --- | --- |
| payment-api-token | very high |
| checkout-api Deployment | high |
| checkout-sa ServiceAccount | high |
| Registry Image | high |
| GitOps Repository | very high |
| Ingress/Gateway Route | medium to high |
## Trust Boundaries
- Internet to Ingress/Gateway
- App Namespace to Kubernetes API
- Pod to Secret Store
- CI to Registry
- GitOps Controller to Cluster
- Namespace to other Namespaces
## Abuse scenarios
| Scenario | Risk | Control |
| --- | --- | --- |
| Pod reads Secret and exfiltrates token | payment-data access | RBAC, minimize Secret mount, NetworkPolicy, rotation |
| ServiceAccount can list Secrets | Secret exfiltration | Least Privilege, `can-i`, audit logs |
| unsigned image is deployed | malware | signature verification, digest pinning, Admission |
| missing NetworkPolicy allows lateral movement | spread | Default Deny, targeted Egress rules |
| CI has cluster-admin | pipeline compromise becomes cluster compromise | GitOps, reduced CI rights |
## Stop criteria
- ServiceAccount can `get/list/watch secrets` without business need.
- Workload runs `privileged` without exception.
- Image is not signed or not scanned.
- Secret rotation is not possible.
- no evidence for Policy exceptions.
Solution 2: RBAC audit
ServiceAccounts:
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-sa
namespace: workshop-security
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: overprivileged-sa
namespace: workshop-security
Minimal Role:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: configmap-reader
namespace: workshop-security
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-sa-configmap-reader
namespace: workshop-security
subjects:
- kind: ServiceAccount
name: app-sa
namespace: workshop-security
roleRef:
kind: Role
name: configmap-reader
apiGroup: rbac.authorization.k8s.io
Intentionally risky Role:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: risky-role
namespace: workshop-security
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["rolebindings"]
verbs: ["create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: risky-binding
namespace: workshop-security
subjects:
- kind: ServiceAccount
name: overprivileged-sa
namespace: workshop-security
roleRef:
kind: Role
name: risky-role
apiGroup: rbac.authorization.k8s.io
Check:
kubectl apply -f rbac.yml
kubectl auth can-i get configmaps --as system:serviceaccount:workshop-security:app-sa -n workshop-security
kubectl auth can-i get secrets --as system:serviceaccount:workshop-security:app-sa -n workshop-security
kubectl auth can-i get secrets --as system:serviceaccount:workshop-security:overprivileged-sa -n workshop-security
kubectl auth can-i create rolebindings --as system:serviceaccount:workshop-security:overprivileged-sa -n workshop-security
Assessment:
app-sais acceptable if the app needs to read ConfigMaps.overprivileged-sais critical because Secret read access and RoleBinding creation open escalation paths.- Fix: remove Secret access, remove RoleBinding creation, use separate break-glass processes.
Solution 3: ServiceAccount token automount
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: no-api-app
namespace: workshop-security
spec:
replicas: 1
selector:
matchLabels:
app: no-api-app
template:
metadata:
labels:
app: no-api-app
spec:
automountServiceAccountToken: false
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
Check:
kubectl apply -f no-api-app.yml
kubectl rollout status deployment/no-api-app
POD=$(kubectl get pod -l app=no-api-app -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -- ls -la /var/run/secrets/kubernetes.io/serviceaccount || true
Expected result:
The path does not exist or contains no automatically mounted token.
Technical decision:
- Pod needs token: Kubernetes API access is technically required, for example for controllers.
- Pod does not need token: normal web app without cluster API usage.
Solution 4: Pod Security Restricted
Label Namespace:
kubectl label namespace workshop-security \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/warn=restricted --overwrite
Insecure Pod:
apiVersion: v1
kind: Pod
metadata:
name: privileged-pod
namespace: workshop-security
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
securityContext:
privileged: true
Expected result:
kubectl apply -f privileged-pod.yml
The API Server rejects the Pod because of Pod Security Admission.
Hardened Pod:
apiVersion: v1
kind: Pod
metadata:
name: restricted-pod
namespace: workshop-security
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
securityContext:
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Check:
kubectl apply -f restricted-pod.yml
kubectl get pod restricted-pod
kubectl describe pod restricted-pod
Solution 5: Workload hardening
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hardened-app
namespace: workshop-security
spec:
replicas: 1
selector:
matchLabels:
app: hardened-app
template:
metadata:
labels:
app: hardened-app
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumes:
- name: tmp
emptyDir: {}
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "while true; do echo ok > /tmp/status; sleep 60; done"]
volumeMounts:
- name: tmp
mountPath: /tmp
securityContext:
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
Check:
kubectl apply -f hardened-app.yml
kubectl rollout status deployment/hardened-app
kubectl get pod -l app=hardened-app -o yaml | grep -E 'runAsNonRoot|allowPrivilegeEscalation|readOnlyRootFilesystem|seccompProfile|capabilities' -A3
Explanation:
readOnlyRootFilesystemrequires a separate writable directory if the app needs to write.drop: ALLreduces Kernel capabilities.allowPrivilegeEscalation: falseprevents certain setuid/setcap escalation paths.
Solution 6: NetworkPolicy
Default Deny:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: workshop-security
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Allow DNS:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: workshop-security
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Frontend to backend:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: workshop-security
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-egress-to-backend
namespace: workshop-security
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 8080
Production follow-up questions:
- Do Pods need Kubernetes API Egress?
- Do Pods need registry, monitoring, or logging Egress?
- Are there external APIs with fixed targets?
- Is IPv6/dual-stack considered?
- Does the CNI actually enforce these Policies?
Solution 7: Supply-chain security
Sample assessment:
# Supply-Chain Review
## Image
- Image: registry.example.com/shop/checkout-api:1.4.2
- Digest: sha256:<digest>
- Pull Policy: IfNotPresent or Always according to platform standard
- Registry: private, approved
- Pull Secret: Namespace-specific, rotated
## Checks
| Check | Result | Decision |
| --- | --- | --- |
| Tag is not latest | fulfilled | ok |
| Digest known | fulfilled | ok |
| CVE scan | 1 High, no Critical | remediation required |
| SBOM present | yes | ok |
| Signature | valid | ok |
| Provenance | CI Build ID present | ok |
## Stop criteria
- `latest` without digest in prod.
- Image from unapproved registry.
- Critical CVE without exception.
- missing signature for production workloads.
- Pull Secret without owner or rotation.
Optional:
trivy image registry.example.com/shop/checkout-api:1.4.2
cosign verify registry.example.com/shop/checkout-api@sha256:<digest>
Solution 8: Admission Policy
Kyverno example: Required Labels
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-standard-labels
spec:
validationFailureAction: Enforce
background: true
rules:
- name: require-owner-environment-cost-center
match:
any:
- resources:
kinds:
- Pod
- Deployment
validate:
message: "owner, environment, and cost-center Labels are required."
pattern:
metadata:
labels:
owner: "?*"
environment: "?*"
cost-center: "?*"
Kyverno example: Ban privileged
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-privileged
spec:
validationFailureAction: Enforce
background: true
rules:
- name: privileged-containers
match:
any:
- resources:
kinds:
- Pod
validate:
message: "Privileged containers are not allowed."
pattern:
spec:
=(containers):
- =(securityContext):
=(privileged): "false"
Operating path:
- Test Policy in Audit/Warn mode.
- Clean up findings.
- Document exceptions with owner and expiration date.
- Activate Enforce.
- Define rollback path for Policy failure.
Solution 9: Secrets security
Sample review:
# Secrets Review
## Inventory
| Secret | Namespace | Usage | Owner | Rotation |
| --- | --- | --- | --- | --- |
| payment-api-token | shop-prod | checkout-api | payments-team | every 90 days |
## Access
- Only checkout-sa may mount the Secret.
- No list rights on Secrets for app ServiceAccounts.
- No Secret in logs, Events, or images.
## Risk
Kubernetes Secrets are Base64-encoded API objects. Protection depends on RBAC, etcd encryption, backup protection, and Pod access.
## Recommended target state
- External Secrets or CSI Secret Store for external Secret source.
- Kubernetes Secret only as short-lived runtime projection.
- Rotation tested.
- Access through ServiceAccount minimized.
## Leak response
1. Identify Secret usage.
2. Isolate affected Pods.
3. Rotate Secret in source system.
4. Update Kubernetes Secret.
5. Restart Pods.
6. Check logs, Events, and audit.
7. Tighten RBAC and NetworkPolicy.
Solution 10: Runtime detection
Sample signal matrix:
| Signal | Risk | Data source | First reaction | Evidence |
|---|---|---|---|---|
| unexpected shell | interactive compromise | Falco/Tetragon, audit exec |
isolate Pod, check audit | process, user, Pod, time |
write access /etc |
manipulation in container | runtime sensor | check workload hardening | file path, process |
| unexpected Egress | exfiltration or C2 | flow logs, CNI, runtime | block Egress | target IP, DNS, Pod |
| Kubernetes API access | token abuse | audit logs | check ServiceAccount | verb, resource, subject |
suspicious kubectl exec |
manual access | API audit | check user and change | user, Pod, command time |
| OOMKill | resource or attack signal | kubelet/Event/metrics | check logs and limits | Events, container status |
Important classification:
Detection does not replace prevention. It complements Pod Security, RBAC, NetworkPolicy, and Admission Control.
Solution 11: Compliance Evidence
Sample finding:
# Compliance Evidence
## Finding 1: Namespace without Pod Security Labels
- Timestamp: 2026-06-12 10:00
- Resource: namespace/team-legacy
- Risk: Insecure Pods can be created without warning or rejection.
- Evidence: `kubectl get ns team-legacy --show-labels`
- Owner: team-legacy
- Fix: Set Labels for enforce/audit/warn.
- Deadline: 14 days
## Finding 2: ServiceAccount with Secret read rights
- Resource: rolebinding/risky-binding
- Risk: Secret exfiltration.
- Evidence: `kubectl auth can-i get secrets --as system:serviceaccount:team-a:app-sa -n team-a`
- Fix: Remove Secret rights, check specific Secret mounting.
## Finding 3: Pod without requests/limits
- Risk: unreliable scheduling, noisy-neighbor effect.
- Fix: Add requests/limits through manifest and Admission.
## Finding 4: Secret without rotation note
- Risk: Long-lived Secret increases damage in case of leak.
- Fix: Document owner, rotation date, and source system.
## Finding 5: Workload without NetworkPolicy
- Risk: Lateral movement.
- Fix: Default Deny and targeted Allow Policies.
Good evidence contains not only a screenshot, but reproducible commands, owner, risk, deadline, and retest.
Solution 12: Incident runbook
Sample runbook:
# Incident Runbook: compromised Pod and Secret leak
## 1. Triage
- Identify signal source: runtime alert, audit log, user report, SIEM.
- Determine affected resource: Namespace, Pod, Node, ServiceAccount, Secret, image.
- Set criticality.
## 2. Isolation
- Tighten NetworkPolicy or isolate Namespace.
- Do not delete Pod too quickly if evidence is required.
- If Node is suspected: cordon/drain Node according to evidence plan.
## 3. Preserve evidence
- Pod YAML
- Events
- Logs and previous logs
- Image digest
- ServiceAccount and RBAC
- affected Secrets
- audit logs
- runtime alerts
Example:
kubectl get pod <pod> -n <ns> -o yaml > evidence-pod.yaml
kubectl describe pod <pod> -n <ns> > evidence-describe.txt
kubectl logs <pod> -n <ns> --all-containers > evidence-logs.txt
kubectl get events -n <ns> --sort-by=.metadata.creationTimestamp > evidence-events.txt
## 4. Containment
- Reduce ServiceAccount rights.
- Rotate tokens and Secrets.
- Block Egress.
- Block insecure image.
## 5. Recovery
- Roll out workload from clean image.
- Check GitOps state.
- Check CI/CD and registry.
- Run smoke tests.
## 6. Lessons learned
- Which Policy would have prevented the incident?
- Which detection worked?
- Which evidence was missing?
- Which RBAC or Secret control will be tightened?
Compromised Pod – minimal commands:
kubectl get pod <pod> -n <ns> -o wide
kubectl get pod <pod> -n <ns> -o yaml
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --all-containers
kubectl logs <pod> -n <ns> --all-containers --previous
kubectl get sa,role,rolebinding -n <ns>
kubectl auth can-i get secrets --as system:serviceaccount:<ns>:<sa> -n <ns>
Secret leak – actions:
- Find Secret usage.
- Block access or isolate workload.
- Rotate Secret in source system.
- Update Kubernetes Secret.
- Restart Pods.
- Revoke old tokens.
- Check logs and audit for use.
- Correct RBAC and Secret process.
Cleanup
kubectl delete namespace workshop-security
Assessment checklist
| Criterion | Completed? |
|---|---|
| Threat Model with assets and trust boundaries | ☐ |
| RBAC escalation paths identified | ☐ |
| ServiceAccount token automount reduced | ☐ |
| Pod Security Restricted applied | ☐ |
| SecurityContext fully hardened | ☐ |
| NetworkPolicy with DNS exception and app allow | ☐ |
| Supply-chain review with stop criteria | ☐ |
| Admission Policy with operating mode | ☐ |
| Secret rotation and leak path documented | ☐ |
| Runtime Detection translated into response plan | ☐ |
| Compliance Evidence formulated defensibly | ☐ |
| Incident runbook with evidence and recovery | ☐ |