Kubernetes and PromQL

Defaults:

• Set memory request equal to the memory limit.
• Set a CPU request but no CPU limit.
• Spread pods across zones and nodes with topologySpreadConstraints.
• Mount Secrets as directories, never with subPath.
• Page on what users feel; use pod-level metrics for diagnosis.

Requests, limits, and spreading pods

Setting	Rule	Why
Memory	Request = limit	Memory can’t be borrowed back.
CPU	Request, no limit	Pods use idle CPU instead of being throttled.
Spread	One constraint on zone, one on hostname	One pod in every zone and on every node before any doubles up.

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: app
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: app
      containers:
        - name: app
          resources:
            requests:
              cpu: 100m
              memory: 2560Mi
            limits:
              memory: 2560Mi

From the app that was starved of memory its whole life.

Disruption budget

Set minAvailable so node upgrades and scale-downs can’t drain below two running pods.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: app

Mount a secret so updates reach running pods

Mount the directory, never a single file with subPath. Kubernetes never updates subPath mounts when the Secret changes.

volumes:
  - name: db-ca
    secret:
      secretName: app-db
      items:
        - key: ca.pem
          path: ca.pem
containers:
  - name: app
    volumeMounts:
      - name: db-ca
        mountPath: /etc/ssl/db
        readOnly: true

From getting CA certificates through cloud secret managers intact.

Pull a cloud secret in with External Secrets

Use property to pick a field out of a JSON secret; omit it for a single-value secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: cloud-secrets
    kind: ClusterSecretStore
  target:
    name: app-db
    creationPolicy: Owner
  data:
    - secretKey: ca.pem
      remoteRef:
        key: production/app/db
        property: ca_pem

Find what logs won’t tell you

Check Events, not only logs and status. Some controllers report failures only as Events, so Programmed: True and clean logs don’t mean the change worked.

kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp
kubectl get events -n envoy-gateway-system --field-selector reason=SyncLoadBalancerFailed

From migrating 50+ TCP services to Envoy Gateway.

Why did that container die?

Read the last termination reason from the pod status.

kubectl get pod POD -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# OOMKilled

PromQL: diagnostics worth a dashboard

Don’t page on these. Page on what users feel - failed or slow requests - and use these queries to find out why.

# Containers restarting repeatedly
increase(kube_pod_container_status_restarts_total[10m]) >= 3

# Any OOM kill
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# An HPA stuck at its maximum (add `for: 2h` in the alert rule)
kube_horizontalpodautoscaler_status_current_replicas
  == kube_horizontalpodautoscaler_spec_max_replicas

# Memory used as a share of the limit, per container
sum by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
  / sum by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})

# Share of CPU periods throttled (only matters if you set CPU limits)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))