# Platform Blueprint

## Objective

The Kubernetes platform provides standardized environments for 12 product teams and 40 to 60 microservices. It should be upgradeable, observable, tenant-capable, recoverable, and cost-aware.

## Cluster Model

| Environment | Model | Rationale |
| --- | --- | --- |
| dev | shared cluster | cost, fast iteration, lower criticality |
| staging | dedicated cluster | production proximity, upgrade and release tests |
| production | dedicated cluster, optionally separated by criticality | blast radius, compliance, stable maintenance windows |

## Availability Zones

Production runs across at least three availability zones. System add-ons and application replicas are distributed across zones. Stateful workloads receive storage classes with suitable zone binding and a documented restore strategy.

## Responsibilities

| Area | Owner | Tasks |
| --- | --- | --- |
| Platform team | platform | clusters, node pools, add-ons, GitOps, Quotas, runbooks |
| App teams | product teams | app manifests, SLOs, requests, probes, functional releases |
| Security | security team | policies, audits, exception processes, compliance evidence |
| Operations/on-call | platform + app | first incident response, escalation, DR exercises |

## Ingress/Gateway

- Central Gateway or Ingress Controller per environment.
- App teams request hostnames and routes via PR.
- TLS certificates are managed through cert-manager or an internal certificate model.
- Wildcard certificates only with documented approval.

## DNS and Certificates

- External DNS records are maintained automatically or through a change process.
- Certificates have owners, expiry monitoring, and a renewal path.
- Production uses separate issuers or PKI policies.

## Storage and Backup

- Default StorageClass for general PVCs.
- Separate StorageClass for stateful/regulated workloads.
- VolumeSnapshots or backup tool for PVC data.
- Restore tests at least quarterly.

## GitOps

- Git is the source of desired state.
- CI creates images and manifest PRs with digests.
- The GitOps Controller synchronizes target Namespaces with limited permissions.
- Rollback is performed by Git revert.

## Observability

- Metrics, logs, traces, and Kubernetes events are collected centrally.
- Platform and app SLOs are documented.
- Alerts point to runbooks.
- Retention is defined by environment and compliance requirements.

## Cost Labels and Showback

Required Labels:

- `owner`
- `environment`
- `cost-center`
- `app`

Showback is evaluated by Namespace, team, environment, and cost source.

# Operating Model

## Recommendation

Managed Kubernetes is recommended for the scenario, provided that provider limits and compliance requirements are met. Self-managed Kubernetes is justified only for specific on-premises or bare-metal requirements. Hybrid is possible but increases network, governance, and operations effort.

## Decision Matrix

| Criterion | Managed | Self-managed | Hybrid |
| --- | --- | --- | --- |
| Control Plane | operated by provider | operated by internal team | shared |
| Upgrade effort | lower, but add-ons remain internal | high | medium to high |
| Security responsibility | shared | mostly internal | shared and complex |
| Provider dependency | high | low to medium | medium |
| Operational risk | provider limits and abstraction | operations errors, PKI/etcd/HA | integration errors |
| Team capabilities | platform and cloud know-how | deep Kubernetes/etcd/PKI | cloud + on-prem + network |
| Cost model | predictable, provider costs | operations staff + infrastructure | complex model |

## Open Decisions

- Separate regulated production data by cluster or by Namespace with additional controls.
- Choose Gateway API vs. Ingress.
- Select backup tool and restore-test frequency.
- Select policy engine and exception process.

# Node Pool Sizing

| Node pool | Purpose | Characteristics | Risks |
| --- | --- | --- | --- |
| system-pool | platform add-ons | taints, stable on-demand nodes, overhead planned | add-on outage blocks the platform |
| app-general | standard web services | zone-distributed, HPA-capable, rolling-update buffer | overrequesting, Pod density |
| worker-cpu | CPU-intensive workers | autoscaling, optionally Spot with retry/PDB | Spot interruption, cost flapping |
| stateful | PVC-near workloads | stable storage attachment, cautious maintenance | zone binding, restore complexity |

## Sizing Rules

- Plan app requests plus DaemonSet overhead.
- Consider `kube-reserved` and `system-reserved`.
- Plan rollout buffer for new Pods during rolling updates.
- Evaluate at least node failure; for production, evaluate zone failure as well.
- Check Pod density and IP address space.
- Check cloud quotas for compute, IPs, volumes, load balancers, and firewall rules.

## Stop Criteria

- Pods remain `Pending` because of resources.
- Cluster Autoscaler cannot add any more nodes.
- IP address space or cloud quota is exhausted.
- CoreDNS, CNI, or CSI shows a critical error rate.
- System add-ons run into OOMKill or CPU throttling.

# Add-on Matrix

| Add-on | Purpose | Owner | Criticality | Upgrade path | SLO/operating objective | Metrics/events | Backup/restore | Failure impact |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CNI | Pod network and NetworkPolicy | platform | critical | staging before prod | Pod network available | CNI errors, Pod Pending | back up configuration | Pods cannot communicate |
| CSI | storage provisioning | platform | critical for stateful | version matching the cluster | PVC provisioning stable | PVC Pending, attach errors | VolumeSnapshots | stateful apps blocked |
| CoreDNS | service discovery | platform | critical | rolling update | low DNS latency | NXDOMAIN rate, latency | back up ConfigMap | Services not discoverable |
| Gateway/Ingress | HTTP access | platform | high | controlled controller upgrade | 99.9% routing | 5xx, backend health | config from Git | external apps unreachable |
| Metrics Server | Resource Metrics API | platform | medium | cluster-compatible | HPA metrics available | Metrics API errors | no data backup | HPA/kubectl top affected |
| Observability | metrics/logs/traces | platform | high | component by component | signal availability | ingest, query, retention | back up storage/config | blind operation during incident |
| GitOps Controller | desired-state sync | platform | high | staging test | sync reliable | drift, sync errors | Git is source | deployments stop or are wrong |
| Policy Engine | governance | platform + security | high | audit before enforce | safe resources | policy reports | policies from Git | unsafe or blocked deployments |
| Backup tool | restore capability | platform | high | restore test | backup + restore successful | backup status | protect backups themselves | DR impossible |
| cert-manager | certificates | platform | high | check issuers | certificates valid | expiry, failed renewals | protect issuers/Secrets | TLS outages |

# Upgrade Runbook: Kubernetes Minor Upgrade

## 1. Target Version and Current Version

- Current version: v1.x
- Target version: v1.y
- Environments: first dev, then staging, then production

## 2. Version Skew Check

```bash
kubectl version
kubectl get nodes -o wide
```

## 3. API Deprecation Scan

- Scan manifests in Git.
- Render Helm charts with `helm template`.
- Render Kustomize overlays with `kubectl kustomize`.
- Document and migrate old API versions.

## 4. CRD and Webhook Check

```bash
kubectl get crds
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get apiservices
```

## 5. Add-on Compatibility

Check CNI, CSI, CoreDNS, Gateway/Ingress, Metrics Server, observability, GitOps, policy engine, and backup tool against the target version.

## 6. Backup and Restore Point

- etcd/provider snapshot or cluster backup.
- Record GitOps revision.
- PVC snapshots for stateful workloads.
- Validate restore process in staging.

## 7. Staging Test

Perform the complete upgrade in staging, including smoke tests and rollback simulation.

## 8. Control-plane Upgrade

Managed: use the provider mechanism. Self-managed: update API Server, Controller Manager, Scheduler, and etcd according to the operations documentation.

## 9. Add-on Check

After the control-plane upgrade, check add-on status:

```bash
kubectl get pods -A
kubectl get events -A --sort-by=.metadata.creationTimestamp
```

## 10. Node-pool Upgrade

```bash
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
# Update node through provider or operations process
kubectl uncordon <node>
```

## 11. Smoke Tests

- API Server reachable.
- CoreDNS resolves Services.
- Ingress/Gateway routes.
- PVC attach/detach works.
- GitOps synchronizes.
- Critical app SLOs are normal.

## 12. Stop Criteria

- API Server error rate increased or API unstable.
- CNI blocks Pod networking.
- CoreDNS unstable or high DNS error rate.
- CSI cannot attach volumes.
- Admission Webhook blocks normal deployments.
- Critical workloads do not become Ready.
- Deprecation findings are not fixed.
- Restore point is missing or backup failed.

## 13. Recovery Plan

- Stop the change.
- Do not continue updating the affected node pool.
- Roll back add-on version if compatible.
- Reset workloads to a known revision via GitOps.
- In case of cluster failure: activate provider support or restore process.

## 14. Completion Report

- Version before/after.
- Time window.
- Affected components.
- Smoke-test results.
- Open risks.
- Lessons learned.

# Observability and SLO

## App SLO: checkout-api

99.9% of successful `/checkout` requests complete within 300 ms over 30 days.

## SLIs

| SLI | Source |
| --- | --- |
| Success rate | Ingress/app metrics |
| p95/p99 latency | App/Ingress metrics |
| Request rate | App/Ingress |
| Error rate | HTTP 5xx and functional errors |

## Signals

| Signal | Use |
| --- | --- |
| Metrics | SLO, alerts, capacity |
| Logs | root-cause analysis |
| Traces | dependencies and latency |
| Kubernetes events | scheduling, image pull, probe, PVC |
| Dashboards | situational overview |
| Alerts | action required with runbook |

## Alerts

- Error rate > 2% for 10 minutes.
- p95 > 300 ms for 15 minutes.
- Deployment not fully available for 10 minutes.
- HPA at maximum for 15 minutes.

## Platform Signals

| Component | Signal |
| --- | --- |
| API Server | latency, 5xx, request rate |
| CoreDNS | DNS latency, errors, replicas |
| Gateway/Ingress | 5xx, backend health, TLS errors |
| Nodes | Ready, MemoryPressure, DiskPressure, PIDPressure |
| PVC/CSI | Pending PVC, attach/detach errors |
| GitOps | sync status, drift, reconcile errors |

## Retention and On-call

- Metrics: 30 to 90 days per environment.
- Logs: 14 to 30 days; regulated data according to policy.
- Traces: shorter, sampling-based.
- Alerts point to concrete runbooks and escalation paths.

# Autoscaling Review: Queue Worker

## Starting Point

Queue-based workers process jobs with load peaks during the day and low load at night.

## Resources

- CPU request based on p95 usage per job type.
- Memory request based on p95/p99 plus safety margin.
- Set memory limit carefully to avoid OOMKills.

## HPA

HPA can supplement CPU-based scaling but is not ideal for queue length. `maxReplicas` limits cost.

## KEDA

KEDA is useful if queue length or lag is the best load metric. Example strategy:

- `minReplicas`: 1 or 0 if cold start is acceptable.
- `maxReplicas`: budget-dependent.
- Scale by queue length per worker.

## VPA

Use VPA in recommendation mode for rightsizing. Automatic mode only after testing, because it can restart Pods.

## Node Autoscaling

Cluster Autoscaler or Karpenter provisions nodes for Pending Pods. Quotas, IPs, NodeClass, and Spot risks must be limited.

## Risks

| Risk | Control |
| --- | --- |
| flapping | stabilization windows and cooldowns |
| cost increase | maxReplicas, budgets, alerts |
| missing metrics | fallback and alert |
| underrequesting | SLO and OOMKill monitoring |
| slow node provisioning | buffer, priority, pre-warming |

# GitOps and CI/CD Flow

## Flow

1. Code commit.
2. CI builds image.
3. CI runs tests and scans.
4. CI pushes image to registry.
5. CI determines image digest.
6. CI creates pull request for manifest change.
7. Review checks diff, policies, resources, and security.
8. Merge updates desired state.
9. GitOps Controller synchronizes the cluster.
10. Drift detection reports manual deviations.
11. Rollback is performed by Git revert.

## Identities and Permissions

| Identity | Required permissions | Not allowed |
| --- | --- | --- |
| CI Build Identity | registry push, scan, create PR | permanent cluster-admin permissions |
| Reviewer | Git review and approval | direct cluster changes without review |
| GitOps Controller | limited permissions on target Namespaces | unlimited cluster permissions without scope |
| Platform team | add-ons, policies, cluster resources | unreviewed ad-hoc changes |
| App team | PRs and namespace-scoped app changes | cluster-wide permissions |

# Tenant Standard: Team Namespace

## Namespace

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: team-a-prod
  labels:
    owner: team-a
    environment: prod
    cost-center: cc-1234
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
  annotations:
    platform.example.com/owner-email: team-a@example.com
```

## ResourceQuota

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a-prod
spec:
  hard:
    pods: "30"
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    persistentvolumeclaims: "10"
```

## LimitRange

```yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-a-prod
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        cpu: 500m
        memory: 512Mi
```

## Role and RoleBinding

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-deployer
  namespace: team-a-prod
rules:
  - apiGroups: ["", "apps", "batch", "networking.k8s.io"]
    resources: ["pods", "pods/log", "services", "configmaps", "deployments", "replicasets", "jobs", "cronjobs", "ingresses"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-deployer-binding
  namespace: team-a-prod
subjects:
  - kind: Group
    name: team-a-developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: app-deployer
  apiGroup: rbac.authorization.k8s.io
```

## Default-deny NetworkPolicy

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
```

## Boundary Between Professional and Advanced + Security

Professional designs tenant governance and standard resources. Advanced + Security additionally checks strict RBAC escalation paths, Admission Enforcement, Secret flows, runtime detection, and incident response.

# Operator Review: database-operator

## Purpose

The hypothetical `database-operator` manages database instances through Custom Resources.

## CRDs

- `databases.example.com`
- `databasebackups.example.com`
- `databaserestores.example.com`

## Permissions

Requires access to its own CRDs, StatefulSets, Services, PVCs, Secrets, and events. Cluster-wide permissions must be justified and minimized.

## Conditions and Events

- `Ready`
- `Progressing`
- `Degraded`
- `BackupReady`
- `RestoreFailed`

## Backup Relevance

CRs alone are not sufficient. PVC data, Secrets, operator configuration, and external backups must be included.

## Upgrade Risks

- CRD schema changes.
- Conversion Webhooks.
- Finalizers block deletion.
- Operator version not compatible.
- Old CRs cannot be migrated.

## Uninstall Risks

- Deleting CRDs also deletes Custom Resources.
- Finalizers can remain stuck.
- PVCs and Secrets can be orphaned.

## Tenant Capability

Prefer permissions limited per Namespace. Use ClusterRole only if technically required.

## Stop Criteria

- ClusterRole with `*` without justification.
- No restore test.
- No CRD versioning strategy.
- No uninstall instructions.
- No owners or SLOs.

# Disaster Recovery Runbook

## Scenario 1: Restore of a Namespace

### Data Classes

- Git state for manifests.
- Secrets from Secret management.
- PVC data from VolumeSnapshots or backup.
- Custom Resources if CRDs are involved.

### Restore Order

1. Create target Namespace.
2. Apply Quotas, RBAC, NetworkPolicies, and Pod Security.
3. Restore Secrets.
4. Restore PVCs or VolumeSnapshots.
5. Synchronize workloads through GitOps.
6. Check Service, DNS, and Gateway paths.
7. Perform smoke tests.

### Validation

- Pods Ready.
- Services with Endpoints.
- App smoke test successful.
- PVC data readable.

### RTO/RPO Assumption

- RTO: 2 hours for a standard Namespace.
- RPO: 24 hours, depending on backup frequency.

## Scenario 2: Restore of an Entire Cluster

### Data Classes

- GitOps repositories.
- Cluster add-ons.
- etcd/cluster state.
- CRDs and Custom Resources.
- Secrets.
- PVC data.
- External systems: DNS, registry, identity provider, databases.

### Restore Order

1. Provision a new cluster or perform etcd restore.
2. Validate CNI, CoreDNS, and CSI.
3. Install CRDs.
4. Install GitOps Controller.
5. Synchronize platform add-ons.
6. Synchronize tenant Namespaces.
7. Restore Secrets and PVC data.
8. Check Gateway/Ingress and DNS.
9. Perform smoke tests and SLO checks.

### Evidence

- Backup ID.
- Git revision.
- Restore start/end.
- Smoke-test outputs.
- Deviations and follow-up work.

# Cost Analysis

## Cost Sources

| Source | Measurement point | Measure |
| --- | --- | --- |
| CPU/memory overrequesting | requests vs. actual usage | p95/p99-based rightsizing |
| idle workloads | replicas without traffic | dev schedules, evaluate scale-to-zero |
| PVCs | PVC without active Pods | check owner, snapshot, delete |
| load balancers | number of external LBs | use shared Gateway/Ingress |
| egress | cloud billing, flow logs | optimize data paths |
| observability | log/metric volume | adjust retention and sampling |

## Risks of Underrequesting

Requests that are too low can worsen scheduling, HPA, stability, and latency. Rightsizing means realistic values, not minimal values.

## Showback Model

Costs are evaluated by Namespace Labels `owner`, `cost-center`, `environment`, and `app`.

## Action Plan

| Priority | Measure |
| --- | --- |
| 1 | introduce required Labels and cost reports |
| 2 | review top Namespaces by overrequesting |
| 3 | clean up unused PVCs and load balancers |
| 4 | scale dev workloads on a schedule |
| 5 | evaluate Spot for fault-tolerant workers |

# Kubernetes Platform Operations Handover

## Platform Overview

- Model: managed Kubernetes.
- Environments: dev, staging, production.
- GitOps: central controller per environment.
- Gateway/Ingress: centrally managed.
- Observability: metrics, logs, traces, events.

## Critical Add-ons

| Add-on | Owner | Runbook |
| --- | --- | --- |
| CNI | platform | runbooks/network.md |
| CoreDNS | platform | runbooks/coredns.md |
| CSI | platform | runbooks/storage.md |
| GitOps | platform | runbooks/gitops.md |
| Gateway/Ingress | platform | runbooks/ingress.md |

## Known SLOs

- API Server and platform signals according to platform SLO.
- App SLOs per product team.

## Alerting Paths

- Critical platform alerts go to platform on-call.
- App alerts go to app on-call with platform escalation.

## Upgrade Windows

- Minor upgrades quarterly.
- Add-on upgrades after staging test.

## Backup and Restore

- Namespace restore and cluster restore documented.
- Restore test at least quarterly.

## Top Five Runbooks

1. CoreDNS outage.
2. CNI/NetworkPolicy outage.
3. CSI/PVC Pending.
4. GitOps sync failure.
5. Kubernetes minor upgrade.

## Stop Criteria

- API Server is not stable.
- CNI or CoreDNS critical.
- CSI cannot provision volumes.
- GitOps synchronizes the wrong desired state.
- Backup or restore point missing before critical change.

## Open Risks

- Some legacy Namespaces without cost Labels.
- Cluster restore test older than 90 days.
- Two add-ons without documented SLO.

03-professional-submission/
  answers.md
  platform-blueprint.md
  operating-model.md
  node-pool-sizing.md
  addon-matrix.md
  upgrade-runbook.md
  observability-slo.md
  autoscaling-review.md
  gitops-flow.md
  tenant-standard.md
  operator-review.md
  dr-runbook.md
  cost-review.md
  handover.md