Nodes: kubelet, containers and cgroups

Key points:

A container is a process with namespaces (what it sees) and cgroups (what it uses).
Over the memory limit: killed (OOMKilled, exit 137). Over the CPU limit: throttled.
Never point a liveness probe at a dependency.
Endpoint removal and SIGTERM happen at once - delay shutdown with a preStop hook.
Pods without requests are BestEffort: evicted first, invisible to the scheduler.

A container is a process with limits

The kernel has no “container” object - just a process started with:

Namespaces - what it sees: process list (pid), network stack (net), mounts (mnt), hostname (uts), IPC, optionally user IDs (user).
cgroups - what it uses: CPU, memory, process count, I/O.
A root filesystem from the image, plus dropped capabilities and seccomp filters.

flowchart TB
    KUBELET[kubelet] -->|CRI over gRPC| CONTAINERD["containerd / CRI-O<br/>images, snapshots"]
    CONTAINERD --> RUNC["runc<br/>creates namespaces + cgroups,<br/>starts the process"]
    RUNC --> PROC[Your process]

What a pod is

Containers in a pod share the network namespace of a tiny pause container that starts first. So they:

share one IP and talk over localhost
can’t both listen on the same port
can share volumes, but otherwise have separate filesystems

That’s why sidecars (proxies, log shippers) work: same network, same lifecycle.

Requests and limits become cgroups

Setting	What the kernel does
CPU request	Relative weight. Under contention, CPU is shared in proportion to requests
CPU limit	Hard quota per 100 ms period. Use it up and the container is throttled until the next period - even on an idle node
Memory request	Used by the scheduler and eviction ranking. Not enforced by the kernel
Memory limit	Hard cap. Exceed it and the kernel kills the process: `OOMKilled`, exit code 137

Default: set memory limits; many teams skip CPU limits. CPU is compressible - starved means slower. Memory isn’t - over means killed. See the app that was starved of memory its whole life.

QoS classes

Every pod gets a class from its requests and limits:

Class	Rule	Under memory pressure
Guaranteed	Every container has CPU and memory requests equal to limits	Killed last
Burstable	At least one request or limit set, but not Guaranteed	In between
BestEffort	No requests or limits at all	Killed first

kubectl get pod my-pod -o jsonpath='{.status.qosClass}'

Eviction vs OOM kill

OOM kill - the kernel kills a process over its own memory limit, or picks a victim when the node runs out. The container restarts in place.
Eviction - the kubelet sees the node low on memory, disk or PIDs and evicts whole pods before the kernel acts. Order: pods over their requests first, then by priority and by how far over. Evicted pods reschedule elsewhere only if a controller owns them.

Probes

Probe	Question	On failure
Startup	Has it finished starting?	Keep waiting; other probes are paused until it passes
Readiness	Should it get traffic right now?	Removed from Service endpoints. Not restarted
Liveness	Is it stuck beyond repair?	Container restarted

Never point liveness at a dependency. If the database is down and every liveness check calls it, every pod restarts at once and still can’t reach it. Check only the process itself.

How a pod shuts down

sequenceDiagram
    participant API as API server
    participant EP as EndpointSlice controller
    participant K as Kubelet
    participant C as Container
    API->>API: Pod gets deletionTimestamp
    par in parallel
        API->>EP: remove from endpoints
    and
        API->>K: pod terminating
        K->>C: run preStop hook
        K->>C: SIGTERM
    end
    K->>C: SIGKILL after terminationGracePeriodSeconds (30s)

Delay shutdown with a preStop hook and finish in-flight work on SIGTERM. Endpoint removal and SIGTERM happen at the same time, and load balancers and kube-proxy on other nodes notice late. An app that exits immediately fails the requests still arriving.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]   # newer clusters: sleep: {seconds: 10}

preStop time counts toward the grace period - leave room for the app’s own shutdown.

Exit codes and states

You see	Means
Exit code 137	SIGKILL (128 + 9): OOM kill, or grace period ran out
Exit code 143	SIGTERM (128 + 15): normal shutdown
Exit code 1 (or other)	The app failed. Check its logs
CrashLoopBackOff	Keeps crashing. Restarts back off from 10 s, doubling up to 5 minutes
ImagePullBackOff	Wrong image name or tag, or missing registry credentials
CreateContainerConfigError	A referenced Secret or ConfigMap key doesn’t exist

kubectl logs my-pod --previous                    # logs from the crashed container
kubectl describe pod my-pod | grep -A5 'Last State'

Stronger isolation

Run untrusted code under another runtime via a RuntimeClass. All containers on a node share one kernel; a kernel exploit escapes them all.

gVisor (runsc) - a user-space kernel intercepts syscalls. Strong isolation, some performance cost, not every syscall supported.
Kata Containers - each pod in a lightweight VM with its own kernel. VM-level isolation, more memory per pod.

Gotchas

CPU throttling at low average usage. The quota is enforced every 100 ms, so a container averaging 20% of its limit still gets throttled in bursts. Watch container_cpu_cfs_throttled_periods_total.
Runtimes ignore container limits. Older JVMs size the heap from node memory and get OOM killed; Go needs GOMEMLIMIT to GC before the limit, and before Go 1.25 sized GOMAXPROCS from every node CPU. Set these explicitly.
BestEffort pods go first. Without requests a pod is evicted first and the scheduler thinks it uses nothing. Set requests.
Readiness failing everywhere at once. If every pod fails readiness together, the whole Service leaves rotation. Keep readiness about this pod’s ability to serve.
PID 1 gets no default signal handling. A shell-script entrypoint may never forward SIGTERM, so every shutdown waits the full grace period and ends in SIGKILL. Use exec in scripts, or a tiny init like tini.