Pod networking and CNI

Key points:

• The CNI plugin, not Kubernetes, implements pod networking: one IP per pod, pod-to-pod without NAT.
• Overlay runs anywhere at some overhead; native routing is faster and easier to debug but spends real network addresses.
• Service load balancing is per connection - gRPC and HTTP/2 need client-side balancing or an L7 proxy.
• NetworkPolicy does nothing unless your CNI enforces it (Flannel doesn’t). Test that traffic is blocked.
• On the AWS VPC CNI, size subnets from max pods per node × max nodes.

The model in three rules

Kubernetes sets three rules; a CNI plugin (Container Network Interface) makes them true:

1. Every pod gets its own IP address.
2. Every pod can reach every other pod by that IP, without NAT.
3. Nodes can reach every pod, and pods can reach every node.

On pod start, the kubelet has the runtime create a network namespace and call the CNI plugin, which adds the interface, IP and routes. Overlays, BGP and eBPF are each plugin’s way of satisfying rule 2.

Pod to pod on the same node

Same-node traffic never leaves the machine. Each pod has a veth pair (eth0 inside the pod, the peer in the node’s root namespace), joined by a bridge or per-pod routes.

Pod to pod across nodes

	Overlay (encapsulation)	Native routing
How	Pod packet wrapped in a node-to-node packet (VXLAN or Geneve); the network only sees node IPs	Pod IPs are real network addresses; packets travel unwrapped
Examples	Flannel, Calico in VXLAN mode, Cilium in tunnel mode	AWS VPC CNI (pod IPs from your VPC subnet); Calico with BGP and Cilium in native mode (advertise pod routes)
Use when	It must run anywhere	You want speed and easier debugging
Cost	Some overhead	Pods use up real network addresses

flowchart LR
    subgraph N1["Node A (10.0.1.10)"]
        P1["Pod 1<br/>10.244.1.5"] --> V1[veth] --> R1[Node routing<br/>or bridge]
    end
    subgraph N2["Node B (10.0.2.20)"]
        R2[Node routing<br/>or bridge] --> V2[veth] --> P2["Pod 2<br/>10.244.2.9"]
    end
    R1 -->|"overlay: wrapped in VXLAN 10.0.1.10 → 10.0.2.20<br/>native: sent as-is, network knows the pod route"| R2

Services: a stable address for moving pods

Point clients at a Service, never at pods - pods come and go. Its ClusterIP is a virtual IP no interface owns; kube-proxy (iptables or IPVS rules) or an eBPF dataplane like Cilium rewrites packets to one of the ready pods in its EndpointSlices.

CoreDNS names each Service my-svc.my-namespace.svc.cluster.local.

Service type	What it gives you
ClusterIP	A virtual IP reachable inside the cluster
NodePort	ClusterIP, plus a port opened on every node
LoadBalancer	NodePort, plus a cloud load balancer in front
Headless (clusterIP: None)	No virtual IP; DNS returns the pod IPs directly. StatefulSets use this for stable per-pod names

Full comparison, EndpointSlices, traffic policies and exposure risks: Services and EndpointSlices.

Gotchas

• AWS VPC CNI runs out of IPs. Every pod takes a VPC address and each node holds a warm pool, so a /24 runs out fast (pods per node are also capped by the instance type’s interfaces and IPs). Size subnets from max pods per node × max nodes, or fix later with prefix delegation, a secondary CIDR with custom networking, or IPv6.
• Load balancing is per connection, not per request. kube-proxy picks a pod per connection, so long-lived gRPC and HTTP/2 connections pin a client to one pod. Use client-side balancing or an L7 proxy.
• NetworkPolicy is silently ignored. A non-enforcing CNI (Flannel) still accepts the policy. Test that traffic is blocked.
• ndots:5 slows external lookups. Names with under five dots try every search domain first, so api.example.com fails several lookups first. Use a trailing dot (api.example.com.) or lower ndots in the pod’s dnsConfig.
• Overlay MTU drops large packets. VXLAN adds 50 bytes of headers, so pings succeed and big responses hang. Lower the pod MTU to match.
• externalTrafficPolicy: Cluster hides the client IP. It can also add a node-to-node hop. Use Local to keep the IP and skip the hop, and spread pods evenly since only nodes with a pod get traffic.
• conntrack table fills up. Each kube-proxy connection uses a conntrack entry; heavy churn fills the table and drops new connections. Look for nf_conntrack: table full in dmesg.

Debugging commands

kubectl get endpointslices -l kubernetes.io/service-name=my-svc   # does the Service have ready pods?
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash          # a pod full of network tools
dig my-svc.my-namespace.svc.cluster.local                           # inside that pod: DNS
curl -v http://my-svc.my-namespace:8080/                            # inside that pod: the Service
kubectl get networkpolicy -A                                        # anything blocking traffic?