Service meshes | infrascribbles

Key points:

Adopt a mesh for per-request mTLS identity and authorization between services; most other needs have cheaper answers.
Default to Linkerd for the simplest mTLS and metrics; pick Istio (ambient first) for depth; pick Cilium’s mesh if you already run Cilium.
Roll out mTLS as PERMISSIVE, verify, then STRICT - STRICT cuts off everything outside the mesh.
Retry in one layer only, and nest timeouts so outer ones are longer than inner ones.
Treat the control plane as tier-0: run it highly available and monitor certificate expiry.

What a mesh gives you

A mesh moves networking out of app code into infrastructure:

Mutual TLS everywhere - every service-to-service connection encrypted, both sides authenticated with auto-rotating certificates.
Identity-based authorization - “only checkout may call POST /charge on payments”, enforced on identity, not IP.
Traffic control - timeouts, retries, circuit breaking, weighted splits for canaries.
Uniform telemetry - request rate, errors and latency per service, no code changes.

Do you need one?

Check the cheaper option for each need first - a mesh is a significant system to run.

Need	Without a mesh
Encrypt traffic between nodes	Cilium or Calico WireGuard encryption
Restrict which pods can talk	NetworkPolicy (identity-based with Cilium)
North-south routing, TLS, rate limits	An ingress or Gateway API controller (Envoy Gateway)
Retries and timeouts	Client libraries, gRPC’s built-in retry policy
Per-request mTLS identity and authorization between services	This is what a mesh is for
Golden metrics for every service without instrumenting each one	Also a mesh strength

Use when: compliance requires mTLS between services, you need zero-trust authorization across many teams, or many services in many languages make consistent in-code retries and telemetry unrealistic.

Architectures

Sidecar (Istio with Envoy, Linkerd with its own Rust micro-proxy) - a proxy container in every pod. Full L7 per pod; costs a proxy’s CPU and memory in every pod.
Ambient / sidecarless (Istio) - a per-node ztunnel does mTLS and L4; optional per-namespace waypoint proxies add L7 where needed. No pod restarts to join.
eBPF plus per-node proxy (Cilium) - L3/L4 in the kernel, L7 through a shared Envoy per node.

flowchart LR
    subgraph SC["Sidecar"]
        PA["Pod A<br/>app + proxy"] -->|mTLS| PB["Pod B<br/>proxy + app"]
    end
    subgraph AM["Ambient"]
        QA["Pod A<br/>app only"] --> ZA[ztunnel<br/>node 1]
        ZA -->|mTLS| W["waypoint<br/>(only if L7 needed)"]
        W --> ZB[ztunnel<br/>node 2]
        ZB --> QB["Pod B<br/>app only"]
    end

Every mesh has a control plane (certificates, config) and a data plane (the proxies). Workload identity is usually a SPIFFE ID from the service account, e.g. spiffe://cluster.local/ns/payments/sa/payments.

Choosing

	Istio	Linkerd	Cilium Service Mesh
Data plane	Envoy sidecars, or ambient (ztunnel + waypoints)	Rust micro-proxy sidecars	eBPF + per-node Envoy
Features	The most: L7 routing, authz, egress gateways, multi-cluster	Focused: mTLS, retries, timeouts, metrics, authz	Builds on Cilium networking and policy
Operational weight	Heaviest, though ambient is lighter	Lightest to run	Light if you already run Cilium
Releases	Open source stable releases	Open source ships edge releases; stable releases come from vendors since February 2024	Part of Cilium releases

Default: Linkerd - the simplest path to mTLS and metrics.
Use Istio when: you need its depth. Start with ambient for lower overhead.
Use Cilium’s mesh when: you already run Cilium and mainly need L7 visibility and policy.

The core configuration (Istio)

Require mTLS mesh-wide, then allow only what’s needed:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system      # root namespace = mesh-wide
spec:
  mtls:
    mode: STRICT               # migrate from PERMISSIVE first
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-checkout
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/checkout/sa/checkout"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charge"]

Set timeouts and bounded retries:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
  namespace: payments
spec:
  hosts: [payments]
  http:
    - route:
        - destination:
            host: payments
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 500ms
        retryOn: connect-failure,refused-stream,unavailable,503

Gotchas

STRICT mTLS breaks everything outside the mesh. Ingress controllers, Prometheus scrapers and un-meshed jobs present no mesh certificate and get cut off. Run PERMISSIVE, confirm in telemetry that all traffic is mTLS, then switch.
Startup races. A classic sidecar may start after the app, so the first connections fail. Use native sidecars (GA in 1.33), which start first; in Istio, also see holdApplicationUntilProxyStarts.
Jobs never finish. A classic sidecar outlives the job’s main container, so the pod never completes. Use native sidecars.
Retry storms. App retries 3x and mesh 3x per attempt: one failure becomes 9+ requests on a struggling service. Retry in one layer only, with a budget and backoff.
Timeouts must nest. A 1 s client timeout before a 2 s mesh timeout leaves the mesh working for nobody. Make outer timeouts longer than inner ones.
Protocol detection stalls. Meshes guess protocols, and server-first ones like MySQL and SMTP confuse it. Name Service ports (http-, grpc-, tcp-) or set appProtocol; declare server-first protocols as TCP.
Upgrades restart every pod. Sidecars update only on pod restart. Plan rolling restarts per namespace within the control plane’s version skew limits.
Control plane is tier-0. When down, existing proxies keep last config, but new pods get no certificates or config, and certificates expire within hours to days. Run it highly available; monitor certificate expiry.
Overhead adds up. Every request crosses two proxies: extra milliseconds per hop, tens of MB per sidecar, times every pod. Budget for it.
Long-lived connections don’t rebalance. An L7 proxy balances HTTP/2 and gRPC per request (an advantage), but raw TCP connections stick to one backend. Expect uneven load on raw TCP.
Envoy error codes are cryptic. In a 503, flag UF = upstream connection failed, UH = no healthy upstream, URX = retries exhausted. Learn the access log flags.
Egress is open by default. The mesh secures service-to-service traffic; calls to external APIs go straight out. Route them through an egress gateway and restrict outbound access.
Ambient L7 needs waypoints. Without one, ambient gives only mTLS and L4 authorization. Deploy a waypoint per namespace or service for HTTP routes, retries and L7 policy.
Cilium kube-proxy replacement bypasses sidecars. Without socketLB.hostNamespaceOnly: true, Istio sidecar routing gets bypassed. Set it - see Cilium vs kube-proxy.