Skip to content
Kubelings

Catalog

All 124 lessons in one place — filter by hands-on vs read-only, by type, or by module. Click any row for the full problem statement; expand a module header for what it teaches.

  • 124Lessons
  • 103🖐 Hands-on
  • 21📖 Read-only
  • 21⚡ Real incidents
  • 16☁ iximiuz only
Two kinds of lesson: 🖐 Hands-on — you fix a live cluster; an automated check gates completion (types: Lab, Incident, Drill). 📖 Read-only — a guided runbook for host-level topics; no cluster tasks (type: Reading). ⚡ Real incident — reproduces a cited, real company outage; open the row for the write-up and postmortem. Platform column — where a lesson runs: iximiuz in the browser on iximiuz Labs · kind locally on a kind cluster · runbook nothing to execute, read-only lessons you follow along. A hands-on lesson showing iximiuz alone needs a real machine (node reboot, systemctl, etcd on disk) — it runs on iximiuz Labs only.
Type Mod Lesson Scenario Platform
Reading M1 welcome Welcome to Kubelings▸ runbook
Lab M1 crashloop-triage CrashLoopBackOff: read the logs before you guess▸ iximiuz kind
Lab M1 expose-web Expose a Deployment: your first Service▸ iximiuz kind
Lab M1 selector-mismatch The Service that routes to nothing▸ iximiuz kind
Lab M1 namespace-basics Namespaces: walls, names, and crossing them▸ iximiuz kind
Lab M1 imagepull-backoff ImagePullBackOff: the tag that never existed▸ iximiuz kind
Lab M1 kubectl-detective kubectl detective: find the broken one▸ iximiuz kind
Lab M2 rolling-update Fix the Rolling Update: unsafe maxSurge/maxUnavailable▸ iximiuz kind
Lab M2 daemonset Build a Node-Level Log Collector DaemonSet▸ iximiuz kind
Lab M2 statefulset StatefulSet with Stable Pod Identity + Headless Service▸ iximiuz kind
Lab M2 jobs The Job That Never Finishes▸ iximiuz kind
Lab M2 cronjobs CronJob Pileup: fix the concurrencyPolicy▸ iximiuz kind
Lab M2 hpa Autoscale a Deployment with an HPA (1 → 5)▸ iximiuz kind
Lab M2 oomkill OOMKilled CrashLoop: right-size the memory limit▸ iximiuz kind
Lab M2 liveness-vs-readiness The liveness probe that kills healthy pods▸ iximiuz kind
Lab M2 startup-probe Slow starter vs impatient liveness▸ iximiuz kind
Lab M2 init-containers Stuck at Init: the gate that never opens▸ iximiuz kind
Lab M2 pdb-blocks-drain The drain that never finishes▸ iximiuz kind
Lab M2 qos-classes QoS classes: who gets killed first▸ iximiuz kind
Lab M2 blue-green-canary Blue/green: the deploy you can undo in one second▸ iximiuz kind
Incident M2 incident-cpu-throttling ⚡ Omio CPU throttling: the latency tax nobody sees▸ iximiuz kind
Lab M2 ephemeral-containers No shell, no exec, no problem▸ iximiuz kind
Lab M2 multi-container-patterns Three pods, three broken patterns▸ iximiuz kind
Drill M2 pattern-readiness-flap Drill — the readiness probe that flaps▸ iximiuz kind
Drill M2 pattern-zombie-cronjobs Drill — zombie CronJobs pile up▸ iximiuz kind
Drill M2 pattern-rolling-update-deadlock Drill — the rolling update that deadlocks▸ iximiuz kind
Lab M2 vpa VPA: the recommender that watched the wrong app▸ iximiuz kind
Lab M2 keda-autoscaling KEDA: the ScaledObject that never scaled▸ iximiuz kind
Incident M2 incident-job-restartpolicy ⚡ Universe Incident replay — restartPolicy: Never did not mean never (Universe)▸ iximiuz kind
Lab M3 configmap-wiring CreateContainerConfigError: the key that isn't there▸ iximiuz kind
Lab M3 secret-not-mounted ContainerCreating forever: the Secret that isn't▸ iximiuz kind
Lab M3 pvc-pending PVC Pending: a claim nobody answers▸ iximiuz kind
Lab M3 access-modes One disk, two nodes: the access-mode trap▸ iximiuz kind
Drill M3 pattern-pvc-terminating Drill — the PVC stuck Terminating▸ iximiuz kind
Lab M3 kustomize-overlays Kustomize: one base, many environments, zero drift▸ iximiuz kind
Lab M3 helm-releases Helm: history, rollback, and the failed release▸ iximiuz kind
Drill M3 pattern-ghost-endpoints Drill — ghost endpoints after scale-down▸ iximiuz kind
Drill M3 pattern-secret-not-reloaded Drill — the Secret that was rotated but never reloaded▸ iximiuz kind
Drill M3 pattern-namespace-terminating Drill — the namespace stuck Terminating▸ iximiuz kind
Reading M3 databases-on-k8s Databases on Kubernetes: what a StatefulSet does not give you▸ runbook
Incident M4 incident-dns-ndots ⚡ Zalando Incident replay — the ndots:5 DNS amplifier (Zalando, Jan 2019)▸ iximiuz kind
Lab M4 networkpolicy-blackhole NetworkPolicy blackhole: default-deny ate my traffic▸ iximiuz kind
Lab M4 broken-targetport Connection refused: port vs targetPort▸ iximiuz kind
Lab M4 nodeport-vs-clusterip Service types: open a door to the outside▸ iximiuz kind
Reading M4 incident-conntrack ⚡ loveholidays Incident file — conntrack: the invisible table that fills up▸ runbook
Incident M4 incident-graceful-shutdown ⚡ Ravelin Incident replay — 502s on every deploy (Ravelin's endpoint secret)▸ iximiuz kind
Lab M4 ingress-wiring Ingress: three names that must agree▸ iximiuz kind
Lab M4 gateway-api Gateway API: routing with a role for everyone▸ iximiuz kind
Reading M4 kube-proxy-dataplane kube-proxy: there is no proxy▸ runbook
Reading M4 cni-basics CNI: who wires the pod, and what breaks when nobody does▸ runbook
Lab M4 kubeconfig-contexts kubeconfig: contexts, the merge, and the prod you almost touched▸ iximiuz kind
Drill M4 pattern-etp-blackhole Drill — externalTrafficPolicy: Local blackholes half your nodes▸ iximiuz kind
Reading M4 incident-ip-exhaustion ⚡ loveholidays Incident file — the cluster that ran out of IP addresses (loveholidays)▸ runbook
Incident M5 incident-same-node ⚡ Moonlight Incident replay — all replicas on the failing node (Moonlight)▸ iximiuz kind
Lab M5 taints-tolerations Everything Pending: who tainted the nodes?▸ iximiuz kind
Lab M5 topology-spread Topology spread: balance, not just separation▸ iximiuz kind
Incident M5 incident-priority-preemption ⚡ Grafana Labs Incident replay — the priority that ate production (Grafana Labs)▸ iximiuz kind
Drill M5 pattern-noisy-neighbor Drill — the noisy neighbor▸ iximiuz kind
Lab M6 rbac-least-privilege cluster-admin for a bot: scope it down▸ iximiuz kind
Incident M6 incident-cryptominer ⚡ JW Player Incident replay — the exposed dashboard (JW Player's cryptominer)▸ iximiuz kind
Incident M6 incident-webhook-outage ⚡ Jetstack Incident replay — the webhook that froze the cluster (Jetstack)▸ iximiuz kind
Lab M6 pod-security-baseline Pod Security: the privileged pod that shouldn't exist▸ iximiuz kind
Lab M6 serviceaccount-tokens The token in every pod (and who's using yours)▸ iximiuz kind
Lab M6 container-hardening Hardening: take away everything the workload doesn't use▸ iximiuz kind
Lab M6 cis-kube-bench Audit the cluster: CIS benchmark with kube-bench▸ iximiuz kind
Reading M6 control-plane-hardening The hardening you can't kubectl: control plane, supply chain, runtime▸ runbook
Lab M6 egress-lockdown Egress lockdown: the miner needs a phone line▸ iximiuz kind
Lab M6 image-digests Tags lie, digests don't: pin the supply chain▸ iximiuz kind
Lab M6 opa-gatekeeper Gatekeeper: the policy that rejected nothing▸ iximiuz kind
Lab M6 kyverno-policies Kyverno: the policy that blocked kube-system▸ iximiuz kind
Lab M6 image-scanning-pipeline Scan it, then pin it: trivy and the digest▸ iximiuz kind
Lab M6 sbom-cosign Signatures and SBOMs: trust, but verify with cosign▸ iximiuz kind
Lab M6 apparmor-seccomp seccomp on, AppArmor understood▸ iximiuz kind
Lab M6 encryption-at-rest Encryption at rest: the full runbook▸ iximiuz
Lab M6 audit-policy Audit policy: who touched that Secret?▸ iximiuz
Lab M6 falco-runtime-detection Falco: alarms for the attack you didn't prevent▸ iximiuz
Reading M6 incident-iam-sidecar ⚡ Adevinta Incident file — the identity sidecar that added a zero to latency (Adevinta)▸ runbook
Lab M7 reconcile-loop The reconcile loop: why deleted pods come back▸ iximiuz kind
Lab M7 scheduler-nodename What the scheduler actually does (bypass it to find out)▸ iximiuz kind
Lab M7 etcd-truth etcd: the one source of truth▸ iximiuz kind
Reading M7 control-plane-tour The control-plane tour: request flow, kubelet, leases▸ runbook
Lab M7 crd-operators CRDs: teach the API server a new noun▸ iximiuz kind
Lab M7 etcd-backup-restore etcd backup & restore: the runbook you rehearse before you need it▸ iximiuz
Lab M7 admission-mutations Admission: the API server edits your YAML before storing it▸ iximiuz kind
Reading M7 watch-informers Watch: how every controller hears the cluster▸ runbook
Reading M7 build-an-operator Build an operator: the verb for your noun▸ runbook
Lab M7 kubeadm-bootstrap kubeadm: a cluster from three commands▸ iximiuz
Lab M7 ha-control-plane HA control plane: three of everything▸ iximiuz
Lab M7 cert-rotation Certificates: the outage scheduled a year in advance▸ iximiuz
Lab M8 events-forensics Events forensics: reconstruct the crime▸ iximiuz kind
Incident M8 incident-node-oom ⚡ Blue Matador Incident replay — the node that OOMed itself (Blue Matador)▸ iximiuz kind
Lab M8 quota-exhausted Deploy blocked: the quota nobody mentioned▸ iximiuz kind
Lab M8 node-notready The morning after NotReady▸ iximiuz kind
Drill M8 pattern-disk-pressure Pattern drill: evicted — the disk you forgot to budget▸ iximiuz kind
Reading M8 incident-datadog-cilium ⚡ Datadog Incident file — the OS under the cluster (Datadog, 2023)▸ runbook
Reading M8 upgrade-runbook The upgrade runbook: version skew, kubeadm, and not becoming a war story▸ runbook
Lab M8 node-maintenance Node maintenance: drain like you mean it▸ iximiuz kind
Lab M8 slo-errorbudget SLOs: the burn-rate alert that never fired▸ iximiuz kind
Lab M8 otel-collector-pipeline OTel pipeline: traces into the void▸ iximiuz kind
Drill M8 pattern-log-pipeline-drop Drill — the log pipeline that ships nothing▸ iximiuz kind
Reading M9 incident-monzo-cascade ⚡ Monzo Incident file — the cascade (Monzo's bank-stopping outage)▸ runbook
Reading M9 incident-openai-cascade ⚡ OpenAI Incident file — locked out of the control plane (OpenAI, 2024)▸ runbook
Reading M9 incident-reddit-piday ⚡ Reddit Incident file — the Pi-Day outage (Reddit, 2023)▸ runbook
Reading M9 incident-black-friday ⚡ Algolia Incident file — killing the dashboard on Black Friday (Algolia)▸ runbook
Reading M9 incident-target-cascade ⚡ Target Incident file — the five-system cascade (Target, 2019)▸ runbook
Reading M9 incident-spotify-delete ⚡ Spotify Incident file — deleting every cluster, twice (Spotify, 2019)▸ runbook
Lab M9 final-boss Final boss: three faults, no hints▸ iximiuz kind
Lab M10 gitops-argocd Argo CD: the app that refuses to sync▸ iximiuz kind
Lab M10 gitops-argocd-appofapps App of apps: one bad child in the fleet▸ iximiuz kind
Lab M10 gitops-flux2 Flux: the Kustomization that can't find its source▸ iximiuz kind
Lab M10 multi-tenancy-capsule Capsule: the tenant that hit its walls▸ iximiuz kind
Reading M10 cluster-api-intro Cluster API: clusters as custom resources▸ runbook
Lab M10 crossplane-compositions Crossplane: the composition missing its provider▸ iximiuz kind
Incident M10 incident-gitops-prune ⚡ Skyscanner Incident replay — a couple of characters pruned the world (Skyscanner)▸ iximiuz kind
Reading M10 incident-istio-integration ⚡ Exponea Incident file — how we failed to integrate Istio (Exponea)▸ runbook
Lab M11 kubelet-down kubelet down: a node goes NotReady from the host up▸ iximiuz
Lab M11 containerd-down containerd down: the CRI is gone and nothing will schedule▸ iximiuz
Lab M11 apiserver-broken-flag you broke the control plane: kube-apiserver in a crash loop▸ iximiuz
Lab M11 etcd-defrag etcd NOSPACE: the alarm that makes your cluster read-only▸ iximiuz
Lab M11 static-pod-create static pods: author one by hand, find its mirror▸ iximiuz
Lab M11 kubelet-csr-approval pending CSRs: approve a node into the cluster▸ iximiuz
Lab M11 cgroup-driver-mismatch cgroup driver mismatch: kubelet and containerd disagree▸ iximiuz
Lab M11 node-clock-skew clock skew: when a node time-travels out of the cluster▸ iximiuz
Lab M11 node-sysctl-networking pod networking dead: the kernel knobs kubeadm needs▸ iximiuz
What finishing this actually gives you (and what it doesn't)

A finisher can honestly claim: production troubleshooting reflexes (the describe → events → ownership-chain ladder, drilled ~60 times), a working internals model (API flow, admission, watch/informers, etcd, scheduler, controllers — including how you'd build one), security posture (RBAC → tokens → admission → hardening → egress → supply chain, plus a real breach walked end-to-end), networking failure literacy (endpoints, DNS, conntrack, the kube-proxy dataplane), supply-chain and policy enforcement (scan → pin → sign → admit, with both major policy engines), platform engineering literacy (GitOps reconciliation, tenancy, infrastructure as Kubernetes APIs), and pattern recognition from 40+ cited real outages. Cert-wise: most of CKA/CKAD, and strong CKS coverage — including the host-execution topics (encryption at rest, audit policy, certificate renewal, etcd restore) as real verify-gated labs on the ☁ iximiuz only track, not just as runbooks you read.

Deliberate non-goals — go elsewhere for these:

  • Executing host-level operations on your own machine — kubeadm upgrades, etcdctl restores, AppArmor profile loading and Falco installs are hands-on, but only on iximiuz Labs, where they run on disposable VMs (the ☁ iximiuz only track). The local runner still never touches your host by design: lesson scripts are confined to the kind node container (Security). The rest stay full runbooks you read.
  • Image building — no container build toolchain in the lesson shell. (Helm is exercised hands-on: lessons that need helm/trivy/cosign install a pinned binary in their init.)
  • Service mesh, multi-cluster federation, cloud-provider specifics (EKS/GKE/AKS quirks) — adjacent ecosystems, not Kubernetes fundamentals. The course tells you when you've reached their doorstep.

Nearly every lesson runs identically on iximiuz Labs and locally on kind from one source of truth; the exception is the ☁ iximiuz only track, whose tasks need a real machine — see Getting Started and Architecture. Single-concept incidents are reproduced inside the module that teaches the concept; multi-concept cascades land in Module 9.

Further reading — 28 cited incidents not (yet) reproduced as labs

Real Kubernetes outages worth studying. These aren't runnable lessons — talks, postmortems and write-ups mapped to the module they most relate to. Rows with a write-up open here; the rest link out.