More Pods Never Helped: The App That Was Starved of Memory Its Whole Life

TL;DR: Took over a customer whose app crashed and slowed down at random. The pods had 0.5 CPU and a 4 GB memory limit. They used about 0.1 CPU and needed about 6.5 GB of memory. Every pod was being OOMKilled, and the autoscaler, which scaled on memory, sat at maximum 24/7 - because adding pods never gives any one pod more memory. Measured for a week, lifted the cap to confirm, then sized from real data: a memory request equal to its limit, a floor of four pods spread across nodes and availability zones with at most two per node, and a PodDisruptionBudget. The crashes and the slowness disappeared. The cluster went from about 15 nodes to 9, and the customer’s EC2 bill dropped 35%.

---

Inheriting a Mystery

An engineer left, and I took over their main customer. I’d touched the system before, but only for small tasks - anything complex went to the owner. So I was learning it almost from zero, and the first thing I learned was that it had a reputation.

The app crashed without anyone getting paged, and every so often it became slow for no obvious reason. Nobody could say why. The autoscaler was always at its maximum, which everyone had come to treat as normal: “it’s a busy app.”

It wasn’t a busy app. It was a hungry one.

Watch a Full Week First

Before changing anything, I watched it for a full week. A week captures the whole cycle - weekday peaks, nightly jobs, quiet weekends. A single day can lie to you.

Three queries told most of the story:

# Memory the container is actually using, against its limit
max by (pod) (container_memory_working_set_bytes{namespace="app", container="app"})

# CPU actually used, in cores
max by (pod) (rate(container_cpu_usage_seconds_total{namespace="app", container="app"}[5m]))

# Why containers last died
kube_pod_container_status_last_terminated_reason{namespace="app", reason="OOMKilled"}

And kubectl confirmed it on any pod with restarts:

kubectl get pod app-7c9f8-xk2lp -n app \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled

Resource	Limit	What the app actually did
CPU	0.5 cores	Never above 0.1
Memory	4 GB	Pinned against the ceiling, then OOMKilled

The memory line was flat against 4 GB in every pod. A container can’t use more memory than its limit - the kernel kills it the moment it tries. So the graph couldn’t show what the app actually wanted. It could only show that it wanted more than it was allowed.

That explained both symptoms:

• The crashes were OOM kills. The container restarted, so the pod “recovered”, and nothing paged.
• The slowness was the app running right against its limit - spending its time on garbage collection and memory pressure instead of work - plus replicas dropping out of service every time one was killed.

Why Scaling Out Couldn’t Help

The autoscaler could scale on CPU or memory. CPU was never the problem, so memory drove it - and it drove it straight to the maximum.

flowchart TD
    A[Pod needs ~6.5 GB<br/>limit is 4 GB] --> B[Memory pinned at limit<br/>utilization ~100%]
    B --> C[HPA: above target<br/>add replicas]
    C --> D[New pod starts<br/>also needs ~6.5 GB]
    D --> B
    B --> E[OOMKilled<br/>restart]
    E --> B
    C --> F[Pending pods<br/>cluster autoscaler adds nodes]
    F --> G[~15 nodes<br/>HPA at max 24/7]

    style A fill:#ff6b6b,color:#fff
    style E fill:#ff6b6b,color:#fff
    style G fill:#ffd93d,color:#000

The HPA measures memory utilization as usage divided by the request. Every pod was at its limit, so every pod looked ~100% utilized. The HPA added a replica, the new replica filled up to 4 GB just like the others, and utilization never came down. Horizontal scaling can’t fix a per-pod limit. Ten pods with 4 GB each are still ten pods that each need 6.5 GB.

Memory-based scaling has a second problem even when the pods are sized right: many runtimes don’t hand memory back to the OS when load drops. Memory goes up with traffic, but it doesn’t come back down, so the HPA scales out and never scales in. Check how your runtime behaves before you trust memory as a scaling signal.

Every replica the HPA added needed a place to run, so the cluster autoscaler added nodes to hold them. We were paying for a fleet of nodes full of pods that were all too small, and for half a core of CPU per pod that sat idle.

The Fix, in Stages

I got the customer’s approval to change it in two stages: first prove the theory, then size for real.

Stage 1: Take the Cap Off and Measure

I turned off the HPA, cut its node group down to 2 nodes, and ran one pod per node with the node’s full resources - a whole r6a.xlarge each, 32 GiB of memory and 4 vCPUs. That’s wildly oversized on purpose. The goal wasn’t efficiency. It was to remove every ceiling and see the app’s real profile for the first time.

With the cap gone, memory climbed past 4 GB and settled at about 6.5 GB. CPU stayed at 0.1. The OOM kills stopped immediately, and so did the slowness. The app had been starved of memory since the day it was deployed.

Turning off autoscaling sounds like a step backwards. It was a diagnostic step. You can’t autoscale an app until you know what one replica actually needs.

Stage 2: Size It From Data

With the real numbers in hand, the final setup: a dedicated node group of r6a.xlarge nodes, pods sized so exactly two fit on a node, and topology spread constraints that put one pod on every node first, then a second.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: app
spec:
  template:
    spec:
      nodeSelector:
        eks.amazonaws.com/nodegroup: app
      topologySpreadConstraints:
        # Spread across availability zones
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: app
        # One pod per node before any node gets a second
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: app
      containers:
        - name: app
          resources:
            requests:
              cpu: 100m        # measured peak ~0.1
              memory: 12Gi     # example: measured ~6.5 GB plus headroom, sized so 2 fit per node
            limits:
              memory: 12Gi     # memory limit = request; no CPU limit
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app
  namespace: app
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: app
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app
  namespace: app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 4
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80   # example

How the pieces fit:

flowchart LR
    subgraph AZa["Zone A"]
        N1["Node 1<br/>pod 1 | pod 5"]
        N2["Node 2<br/>pod 2 | pod 6"]
    end
    subgraph AZb["Zone B"]
        N3["Node 3<br/>pod 3 | pod 7"]
    end
    subgraph AZc["Zone C"]
        N4["Node 4<br/>pod 4 | pod 8"]
    end

    style N1 fill:#96ceb4,color:#000
    style N2 fill:#96ceb4,color:#000
    style N3 fill:#96ceb4,color:#000
    style N4 fill:#96ceb4,color:#000

Pods 1-4 are the floor: one per node. When load rises, the HPA adds pods 5-8, and the hostname spread constraint puts each one in the empty slot on a node that only has one. A few things worth noting:

• Memory request equals the memory limit. CPU gets a request, but no limit. Memory can’t be shared - run out and the kernel kills the process - so the pod reserves exactly what it’s allowed to use and never gets overcommitted. CPU is different: a pod that briefly needs more can borrow idle CPU on the node instead of being throttled, and the request still guarantees it a fair share when the node is busy.
• The memory request decides how many pods fit on a node. An r6a.xlarge has 32 GiB, but after the kubelet, system reservations and DaemonSets, a bit under 29 GiB is allocatable. Two pods fit. A third doesn’t, so the node can never be overpacked. Check your own number with kubectl describe node | grep -A6 Allocatable before you pick a size.
• Scaling up never waits for a node. The second slot on every node is already paid for, so pods 5-8 start in seconds instead of waiting minutes for the cluster autoscaler to provision a node. That headroom is a deliberate cost: at the floor, each node runs half-empty. For an app whose problem was slowness, fast scale-up was worth it.
• Spread across zones and nodes. Losing a zone takes out at most two of the four floor pods. Two pods can carry peak load for a short time - long enough for the HPA to fill the empty slots on the surviving nodes, with no new nodes needed. Check that for your app - it’s the number the whole design rests on.
• The PodDisruptionBudget keeps at least two pods running - the same number we know can carry the load. Node upgrades and scale-downs can’t drain below it.
• The HPA still scales on memory - and now it works. Utilization is usage divided by the request. Before, every pod sat at its limit, so utilization read ~100% no matter how many pods there were. Now a pod’s memory request matches what it really needs, so utilization goes up with load and comes back down when it drops. Same metric, same HPA - it just finally measures something real.

flowchart LR
    subgraph "Before"
        B1[HPA at max 24/7]
        B2[Many pods<br/>0.5 CPU / 4 GB each]
        B3[OOMKilled restarts]
        B4[~15 nodes]
        B1 --> B2 --> B3
        B2 --> B4
    end

    subgraph "After"
        A1[HPA on memory<br/>4 to 8 pods]
        A2[1 pod per node, then 2<br/>sized from real usage]
        A3[PDB: minAvailable 2]
        A4[9 nodes<br/>pods share nodes]
        A1 --> A2 --> A4
        A3 --> A2
    end

    style B3 fill:#ff6b6b,color:#fff
    style B4 fill:#ff6b6b,color:#fff
    style A2 fill:#96ceb4,color:#000
    style A4 fill:#96ceb4,color:#000

Why Fewer Nodes Cost Less

Giving every pod nearly twice the memory and ending up on fewer nodes sounds backwards. It isn’t:

• The autoscaler stopped buying nodes for the wrong problem. Much of that fleet existed to hold replicas the HPA added in response to a per-pod limit. More replicas never helped, so all of that capacity was waste.
• We stopped paying for CPU nobody used. Half a core reserved per pod, with 0.1 used, adds up fast across a maxed-out replica count.
• Every node carries fixed overhead. DaemonSets, system-reserved memory, and leftover fragments too small to fit another pod. Fewer nodes means less of all three.
• r6a is memory-optimized. 32 GiB per 4 vCPUs suits an app that needs memory and almost no CPU.

The Results

Metric	Before	After
Memory per pod	4 GB limit, OOMKilled	Sized from ~6.5 GB measured usage
CPU per pod	0.5 cores (0.1 used)	100m request, no limit
HPA	At max 24/7	4 to 8 pods, scales on memory
Cluster nodes	~15	9
Customer EC2 bill	-	35% lower
Crashes and slowdowns	Random, frequent	Gone

More stable and cheaper at the same time. That’s rare, and it usually means the original setup was wrong, not that you found a clever trick.

The app has changed a lot since then, and its pods have been resized more than once. The method hasn’t changed: measure first, then size.

Alerting on What Users Feel

This app crashed for its whole life without paging anyone. Every crash looked like a pod that restarted and recovered, so nothing inside the cluster ever looked broken for long.

What I added was an external endpoint check with a latency threshold. It runs from outside our infrastructure, the same way a client connects, and alerts on two things:

• The endpoint fails - that’s the crashes, seen the way users see them.
• The endpoint responds, but slowly - that’s the “randomly slow” complaint, which no restart counter would ever catch.

That’s the point of checking from the outside. Pod restarts, OOM kills, and HPA replica counts are causes. A failed or slow request is the symptom your customer actually notices. Alert on the symptom, and use the causes to diagnose.

The in-cluster signals are still worth having as diagnostics - on a dashboard or as tickets, not pages:

# A container restarting repeatedly
increase(kube_pod_container_status_restarts_total[10m]) >= 3

# Any OOM kill at all
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# An HPA sitting at its maximum for hours - a warning sign, not "a busy app"
kube_horizontalpodautoscaler_status_current_replicas
  == kube_horizontalpodautoscaler_spec_max_replicas

Put a for: 2h or longer on the last one. An HPA touching its maximum during a peak is fine. One that lives there is telling you something is wrong.

---

Takeaways

1. Watch a full week before you change anything. One day misses the weekly cycle. A week shows you peaks, jobs, and quiet time.

2. A graph pinned at the limit doesn’t show what the app needs. It shows that it needs more. Lift the cap in a controlled way to see the real number.

3. Horizontal scaling can’t fix a per-pod limit. If every pod is starved, more starved pods won’t help.

4. Memory scaling only works when the request is right. Starved pods look permanently busy, so the HPA can only go to max. Size the pod first, then let memory drive scaling.

5. An HPA permanently at max is a warning sign, not a personality. Put it on a dashboard and look at it.

6. Alert on what users feel. An external endpoint check with a latency threshold catches both crashes and slowness. Restart counts tell you why, not whether anyone noticed.

7. Set memory request equal to the limit; be careful with CPU limits. Memory can’t be borrowed back, CPU can. Size both from measured usage.

8. Fewer, bigger, right-sized pods can cost less. Overhead per node and unused reservations are where the money goes.

9. Inherited systems hide things. The setup everyone had accepted as normal was the bug. Fresh eyes and a week of data found what years of “it’s just busy” didn’t.