From kubectl apply to a running container

Key points:

• Components never call each other - they watch the API server and write results back.
• kubectl apply returns once etcd stores the object. Nothing is running yet.
• etcd needs a quorum to write. Without it nothing changes, but running pods keep running.
• The scheduler sees only requests, never real usage.
• Admission webhooks sit in the hot path of every matching write.

The components

Component	Runs on	Job
kube-apiserver	Control plane	The only front door: authenticates, authorizes, validates and stores every object. Nothing else talks to etcd
etcd	Control plane	Replicated key-value store holding every object
kube-controller-manager	Control plane	Dozens of controllers, each making reality match one kind of object
kube-scheduler	Control plane	Picks a node for every pod without one
kubelet	Every node	Runs the pods assigned to its node
Container runtime	Every node	containerd or CRI-O: pulls images, starts containers
kube-proxy or an eBPF dataplane	Every node	Routes Service traffic to pods

Components don’t call each other. Each watches the API server for its objects, does its part and writes the result back. The API server is the only shared point.

Step by step

sequenceDiagram
    participant U as kubectl
    participant API as API server
    participant E as etcd
    participant DC as Deployment controller
    participant RC as ReplicaSet controller
    participant S as Scheduler
    participant K as Kubelet
    participant R as Runtime (CRI)
    U->>API: apply Deployment
    API->>API: authn, RBAC, admission, validation
    API->>E: store Deployment
    DC-->>API: watch: new Deployment
    DC->>API: create ReplicaSet
    RC-->>API: watch: new ReplicaSet
    RC->>API: create Pods (no node yet)
    S-->>API: watch: unscheduled Pods
    S->>API: bind Pod to node-2
    K-->>API: watch: Pod on node-2
    K->>R: sandbox, pull image, start containers
    K->>API: update Pod status: Running, Ready

1. kubectl computes the change (or leaves it to the server with --server-side) and sends it over HTTPS.
2. API server: authentication (certificate, token or OIDC), RBAC, mutating admission, schema validation, validating admission. See policy as code.
3. etcd stores it. kubectl apply returns here - nothing is running yet.
4. Deployment controller creates a ReplicaSet for the current pod template.
5. ReplicaSet controller wants 3 pods, sees 0, creates 3 Pods with no nodeName.
6. Scheduler filters out unfit nodes, scores the rest, writes a binding to the best.
7. Kubelet calls the runtime via the CRI (Container Runtime Interface): create the pod sandbox (network namespace plus CNI plugin call), pull images, start containers.
8. Kubelet reports status. Once readiness probes pass, the pod is Ready, the EndpointSlice controller adds it to its Services and kube-proxy sends it traffic.

etcd in one minute

• Raft consensus. Writes need a quorum: 2 of 3 members, 3 of 5. Without it nothing changes - existing pods run, nothing new is scheduled.
• Use 3 or 5 members. 4 tolerate the same single failure as 3, with more work per write.
• resourceVersion comes from etcd’s revision counter; watches resume from it. If it was compacted away, the API server returns 410 Gone and the client must list again.
• Size limits: about 1.5 MB per object, 2 GB database quota by default. Huge ConfigMaps and old Events add up.
• Back it up. On self-managed control planes, etcdctl snapshot save is the only real backup. EKS, AKS and GKE handle this for you.

How the scheduler decides

1. Filter - drop nodes where:
   • requested CPU or memory doesn’t fit (requests, not real usage)
   • nodeSelector or node affinity doesn’t match
   • a taint isn’t tolerated
   • the pod’s volume is in another zone
   • topology spread or pod anti-affinity would break
2. Score - spread across zones and nodes, balance resource use, prefer nodes that have the image.
3. Bind - write the choice. If no node passes, the pod stays Pending with an event saying why.

A high-priority pod that can’t fit can preempt (evict) lower-priority pods.

Gotchas

• Apply success means stored, not running. The command returns once etcd has the object. Wait with kubectl rollout status deployment/my-app.
• The scheduler only knows requests. Tiny requests with huge real usage get packed onto a full node, and pods with no requests look free. Set requests that match real usage.
• Pending pods tell you why. The reason is recorded as an event. Read it in kubectl describe pod → Events: 0/9 nodes are available: 3 Insufficient memory, 6 node(s) had untolerated taint.
• Control plane down doesn’t stop running pods. The kubelet and runtime keep containers running. Expect no changes and no rescheduling after a node failure until it’s back.
• Admission webhooks are in the hot path. Every matching write, including from controllers, waits on them. Keep them fast and reliable, or a slow one slows writes and a failing one blocks them.