Production clusters pin all workloads to a single AZ per region (cost: cross-AZ data transfer). If that AZ degrades, the whole cluster degrades with it. This page is the recovery procedure. Expected recovery: 15-45 minutes for stateless workloads once step 3 lands.

Design and per-cluster details: Single AZ Migration (Notion).

1. Confirm it is an AZ failure

  • Nodes NotReady concentrated in one zone: kubectl get nodes -L topology.kubernetes.io/zone
  • AWS Health Dashboard shows the AZ event
  • Pods Pending with no nodes joining

Do not evacuate for a single bad node - recycle it instead (kubectl delete node <node>, Karpenter drains and replaces it).

2. Pick the surviving AZ

Any other AZ of the region, with one hard exception: ap-northeast-3a has no i3en instances - cachey cannot run there, use 3c. Two checks before step 3:

# the AZ offers our instance types (offerings, not live capacity - ICE can
# still happen and Karpenter falls back across the allowed families)
aws ec2 describe-instance-type-offerings --region <region> \
  --location-type availability-zone \
  --filters Name=location,Values=<az> Name=instance-type,Values=i3en.xlarge,r7g.xlarge,m7g.4xlarge

# the AZ's subnet has room for the whole cluster (see appendix for the command)

3. Move the cluster

Edit the region’s Karpenter values in the vega repo - the same one-line change that pinned it:

# infra/helm/apps/devops/karpenter/production/<region>/values.yaml
nodePools:
  zones: ["<surviving-az>"]

Merge fast (emergency review), then sync the karpenter app in ArgoCD. Karpenter marks all nodes as drifted and rebuilds them in the surviving AZ, draining with PDBs. The us-east-2 temporal spoke has its own values file (production/us-east-2/values.temporal.yaml).

Update the quickwit searcher and cachey zone pins in lockstep - they carry hard nodeAffinity to the dead AZ and will stay Pending otherwise: infra/helm/apps/quickwit/production/<region>/values.yaml (+ offline/) and infra/helm/apps/cachey/production/<region>/values.yaml (+ offline/).

4. Kick the stragglers

  • Pods annotated karpenter.sh/do-not-disrupt never move on their own. Discover them (today: cachey, quickwit metastore, trino-gateway) and restart each owning workload:

    kubectl get pods -A -o json | jq -r '.items[]
  | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] != null)
  | [.metadata.namespace, .metadata.ownerReferences[0].name] | @tsv' | sort -u

  • EBS-backed pods are anchored to volumes in the dead AZ. Snapshot-restore each into the surviving AZ (volumes are enumerated per cluster in the design doc; typically 1-5: ai-agents, maki-web, quickwit-indexer, backoffice postgres). Never edit the backyard-postgres volumeClaimTemplate - it is immutable and rejects the whole app sync.

5. What takes care of itself

  • Aurora: fails over to its other-AZ instance automatically (expect up to ~1 min of write errors during the switch). Data is never AZ-local (storage is replicated across three AZs). Verify: aws rds describe-db-clusters shows a writer, and app smoke tests pass.
  • EKS control plane, ALB, RDS Proxy, ElastiCache, OpenSearch: multi-AZ by design; expect brief connection resets while their endpoints re-resolve, then steady state.
  • NAT: lives in AZ-a of every prod VPC. If AZ-a is the one that died, internet egress is down regardless of pods - this predates the single-AZ migration and evacuation does not fix it.

6. Validate 

kubectl get nodes -L topology.kubernetes.io/zone --no-headers \
  | awk '{print $6}' | sort | uniq -c                              # one zone only
kubectl get pods -A --field-selector=status.phase=Pending          # empty
kubectl get pods -A --no-headers | awk '$3=="Running"' \
  | grep -vE '^\S+\s+\S+\s+([0-9]+)/\1\s' | head                   # no 0/N ready

Plus the usual: synthetics green, no new FailedCreatePodSandBox events in Datadog.

7. Return (optional, when the AZ recovers)

Same procedure in reverse. No urgency - the only cost of staying in the fallback AZ is that searcher/cachey caches were rebuilt there.

Appendix: subnet free-IP check

The pinned AZ’s subnet is the cluster’s IP budget. Check it on demand (also do this before adding significant workloads):

aws ec2 describe-subnets --region <region> \
  --filters Name=availability-zone,Values=<az> "Name=tag:Name,Values=*private-subnet*" \
  --query 'Subnets[].[Tags[?Key==`Name`]|[0].Value,CidrBlock,AvailableIpAddressCount]' \
  --output table --profile prod

The percentage is per node subnet: AvailableIpAddressCount divided by the CIDR size minus the 5 AWS-reserved addresses (a /19 has 8,187 usable). Under ~25% free: add another subnet in the same AZ from spare VPC space - additive terraform, zero disruption (additional_private_subnets in terraform-multi-cloud; prod/us-east-1/eks-regional/main.tf is the worked example - the module applies the karpenter discovery and ELB tags and route table association automatically). Same-AZ subnet-to-subnet traffic is free.