Replacing ingress-nginx with Envoy Gateway: Rate Limiting, IP Whitelists, and OAuth2

TL;DR: ingress-nginx is reaching end-of-life. Migrated to Envoy Gateway using the Gateway API. Built a Helm chart that handles per-route rate limiting, IP allowlists, TLS certificates, and could plug in OAuth2 - all as declarative Kubernetes resources instead of scattered annotations. The Gateway API is what Ingress should have been from the start.

---

Why Move

If you’ve been running ingress-nginx, you’ve probably seen the announcement: ingress-nginx reaches end-of-life in March 2026. No more releases, no more security patches.

That alone is reason enough to migrate. But honestly, ingress-nginx had been frustrating me for a while:

Pain Point	What It Looked Like
Annotations everywhere	Rate limits, CORS, rewrites - all crammed into annotations
No per-route policies	Want rate limiting on one route but not another? Good luck
IP whitelisting hacks	nginx.ingress.kubernetes.io/whitelist-source-range on every Ingress
No standard API	Every controller has its own annotation dialect
Auth bolted on	OAuth2 proxy required a separate deployment + annotations to wire it

The Gateway API fixes all of this. Instead of annotations on Ingress resources, you get proper Kubernetes resources for each concern: routing, traffic policy, security policy, TLS. Each one is independently targetable to specific routes.

The Architecture

flowchart TB
    subgraph "Before: ingress-nginx"
        CLIENT1[Client] --> NLB1[NLB]
        NLB1 --> NGINX[ingress-nginx]
        NGINX --> |"Annotations for<br/>everything"| ING1[Ingress Resource]
        ING1 --> SVC1[Service A]
        ING1 --> SVC2[Service B]
    end

    style NGINX fill:#ff6b6b,color:#fff

flowchart TB
    subgraph "After: Envoy Gateway"
        CLIENT2[Client] --> NLB2[NLB]
        NLB2 --> ENVOY[Envoy Proxy]
        ENVOY --> GW[Gateway]
        GW --> ROUTE1[HTTPRoute A]
        GW --> ROUTE2[HTTPRoute B]
        GW --> ROUTE3[GRPCRoute C]
        ROUTE1 --> SVC3[Service A]
        ROUTE2 --> SVC4[Service B]
        ROUTE3 --> SVC5[Service C]

        SP[SecurityPolicy<br/>IP Allowlist] -.-> |"targets"| ROUTE1
        BTP[BackendTrafficPolicy<br/>Rate Limit] -.-> |"targets"| ROUTE1
        CTP[ClientTrafficPolicy<br/>TLS + Proxy Protocol] -.-> |"targets"| GW
    end

    style ENVOY fill:#4ecdc4,color:#000
    style SP fill:#ffd93d,color:#000
    style BTP fill:#45b7d1,color:#000
    style CTP fill:#96ceb4,color:#000

The key difference: policies are separate resources that target specific routes or the gateway itself. No more stuffing everything into annotations.

The Helm Chart

I built a wrapper chart around the upstream Envoy Gateway Helm chart. The upstream chart installs the control plane. My templates handle the Gateway API resources - the actual routing, policies, and certificates.

envoy-gateway/
├── Chart.yaml
├── production.yaml          # Production overrides
├── staging.yaml             # Staging overrides
├── charts/
│   └── gateway-helm/        # Upstream Envoy Gateway (v1.6.3)
└── templates/
    ├── gateway-class.yaml       # GatewayClass
    ├── gateway.yaml             # Gateway with per-app listeners
    ├── envoy-proxy.yaml         # EnvoyProxy data plane config
    ├── client-traffic-policy.yaml  # TLS, proxy protocol
    ├── security-policy.yaml     # IP allowlists
    ├── backend-traffic-policy.yaml # Rate limiting
    ├── certificates.yaml        # cert-manager Certificates
    ├── cluster-issuer.yaml      # Let's Encrypt ACME issuer
    └── reference-grants.yaml    # Cross-namespace route attachment

How the Pieces Fit Together

flowchart TD
    subgraph "Helm Chart Deploys"
        GC[GatewayClass] --> |"references"| EP[EnvoyProxy<br/>NLB config, replicas]
        GW[Gateway] --> |"uses"| GC
        GW --> |"listeners per app"| L1[Listener: app-a.example.com]
        GW --> |"listeners per app"| L2[Listener: app-b.example.com]
        GW --> |"listeners per app"| L3[Listener: api.example.com]

        CERT[Certificate] --> |"per listener"| L1
        CERT2[Certificate] --> |"per listener"| L2
        CERT3[Certificate] --> |"per listener"| L3
        CI[ClusterIssuer<br/>Let's Encrypt] --> CERT
        CI --> CERT2
        CI --> CERT3

        CTP[ClientTrafficPolicy] -.-> |"targets"| GW
        SP[SecurityPolicy] -.-> |"targets routes"| ROUTE1
        BTP[BackendTrafficPolicy] -.-> |"targets routes"| ROUTE1

        RG[ReferenceGrant] --> |"allows cross-ns"| GW
    end

    subgraph "App Namespaces (not in chart)"
        ROUTE1[HTTPRoute] --> |"attaches to"| GW
        ROUTE2[HTTPRoute] --> |"attaches to"| GW
    end

    style GW fill:#4ecdc4,color:#000
    style SP fill:#ffd93d,color:#000
    style BTP fill:#45b7d1,color:#000
    style CTP fill:#96ceb4,color:#000
    style CI fill:#ff6b6b,color:#fff

The chart deploys the infrastructure. Application teams define their own HTTPRoutes in their namespaces. ReferenceGrants allow those routes to attach to the central Gateway.

The Gateway

The Gateway is the core - it replaces the ingress-nginx controller’s listener config. Each application gets its own HTTPS listener with a dedicated TLS certificate:

# templates/gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: {{ .Values.gateway.name }}
spec:
  gatewayClassName: {{ .Values.gateway.className }}
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: {{ .Values.envoyProxy.name }}
  listeners:
    # Generates HTTP + HTTPS listener pairs per app
    {{- range .Values.gateway.listeners }}
    - name: http-{{ .name }}
      protocol: HTTP
      port: 80
      hostname: {{ .hostname }}
    - name: {{ .name }}
      protocol: HTTPS
      port: 443
      hostname: {{ .hostname }}
      tls:
        mode: Terminate
        certificateRefs:
          - name: {{ .certificateRef }}
    {{- end }}

The values file makes adding a new app trivial:

gateway:
  enabled: true
  name: envoy-gateway
  className: envoy-gateway
  listeners:
    - name: https-app-a
      hostname: app-a.example.com
      certificateRef: app-a-tls
      namespace: app-a
    - name: https-app-b
      hostname: app-b.example.com
      certificateRef: app-b-tls
      namespace: app-b
    - name: https-api
      hostname: api.example.com
      certificateRef: api-tls
      namespace: api

Add a listener, add a ReferenceGrant, deploy. The certificate gets auto-generated by cert-manager via the ClusterIssuer.

Certificate Automation

Each listener automatically gets a Let’s Encrypt certificate:

# templates/cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: {{ .Values.clusterIssuer.name }}
spec:
  acme:
    email: {{ .Values.clusterIssuer.email }}
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: {{ .Values.gateway.name }}

# templates/certificates.yaml - one per listener
{{- range .Values.gateway.listeners }}
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: {{ .certificateRef }}
spec:
  secretName: {{ .certificateRef }}
  issuerRef:
    name: {{ $.Values.clusterIssuer.name }}
    kind: ClusterIssuer
  dnsNames:
    - {{ .hostname }}
{{- end }}

The gatewayHTTPRoute solver is key - cert-manager creates temporary HTTPRoutes for the ACME HTTP-01 challenge, attached to the same Gateway. No separate ingress needed.

Rate Limiting

With ingress-nginx, rate limiting was a global annotation:

# Old way: ingress-nginx
metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "5"

With Envoy Gateway, rate limiting is a BackendTrafficPolicy that targets specific routes:

# templates/backend-traffic-policy.yaml
{{- range .Values.backendTrafficPolicies }}
{{- range .targetRoutes }}
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: {{ .name }}-rate-limit
  namespace: {{ .namespace }}
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: {{ .name }}
  rateLimit:
    type: Local
    local:
      rules:
        - limit:
            requests: {{ $policy.rateLimit.requests }}
            unit: {{ $policy.rateLimit.unit }}
{{- end }}
{{- end }}

Values:

backendTrafficPolicies:
  - name: rate-limit
    enabled: true
    targetRoutes:
      - name: app-a
        namespace: app-a
      - name: api-public
        namespace: api
    rateLimit:
      requests: 100
      unit: Minute

Want 100 req/min on your public API but unlimited on internal services? Just don’t add the internal route to the target list. No annotations, no global config.

flowchart LR
    subgraph "Rate Limiting Targets"
        BTP[BackendTrafficPolicy<br/>100 req/min]
        BTP -.-> |"targets"| R1[HTTPRoute: app-a]
        BTP -.-> |"targets"| R2[HTTPRoute: api-public]
        R3[HTTPRoute: internal-api] --> |"no policy<br/>unlimited"| SVC3[Internal Service]
    end

    R1 --> SVC1[App A]
    R2 --> SVC2[Public API]

    style BTP fill:#45b7d1,color:#000
    style R3 fill:#96ceb4,color:#000

IP Whitelisting

This was one of the biggest wins. With ingress-nginx, whitelisting looked like this:

# Old way: copy-paste this onto every Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.1/32,10.0.0.2/32,..."

Miss one Ingress? That route is wide open. Update the IP list? Touch every Ingress resource.

With Envoy Gateway, it’s a SecurityPolicy:

# templates/security-policy.yaml
{{- range .Values.securityPolicies }}
{{- range .targetRoutes }}
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: {{ .name }}-ip-allowlist
  namespace: {{ .namespace }}
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: {{ .name }}
  authorization:
    defaultAction: Deny
    rules:
      - name: allow-whitelisted-ips
        action: Allow
        principal:
          clientCIDRs:
            {{- range $policy.authorization.rules }}
            {{- range .cidrs }}
            - {{ . }}
            {{- end }}
            {{- end }}
{{- end }}
{{- end }}

Values:

securityPolicies:
  - name: ip-allowlist
    enabled: true
    targetRoutes:
      - name: admin-panel
        namespace: admin
      - name: staging-app
        namespace: staging
    authorization:
      defaultAction: Deny
      rules:
        - name: allow-whitelisted-ips
          action: Allow
          cidrs:
            - "203.0.113.10/32"    # Office
            - "198.51.100.0/24"    # VPN range
            - "192.0.2.50/32"      # CI/CD runner

One IP list, applied to multiple routes. Add or remove an IP in one place, redeploy, done.

flowchart TD
    subgraph "SecurityPolicy: IP Allowlist"
        SP[SecurityPolicy<br/>Default: DENY]
        SP --> |"Allow"| CIDR1[Office IPs]
        SP --> |"Allow"| CIDR2[VPN Range]
        SP --> |"Allow"| CIDR3[CI/CD IPs]
    end

    SP -.-> |"targets"| R1[HTTPRoute: admin-panel]
    SP -.-> |"targets"| R2[HTTPRoute: staging-app]

    R3[HTTPRoute: public-site] --> |"no policy<br/>open to all"| SVC[Public Service]

    style SP fill:#ffd93d,color:#000
    style R3 fill:#96ceb4,color:#000

Getting Client IPs Right

One thing that tripped me up: the client IP detection depends on your load balancer setup. If you’re behind an NLB with PROXY protocol, you need a ClientTrafficPolicy:

# templates/client-traffic-policy.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: {{ .Values.gateway.name }}-client-policy
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: {{ .Values.gateway.name }}
  enableProxyProtocol: true       # NLB sends PROXY protocol v2
  clientIPDetection:
    xForwardedFor:
      numTrustedHops: 0           # Direct from NLB, no CDN
  tls:
    minVersion: "1.2"

Behind a CDN like Cloudflare? Set numTrustedHops: 1 and disable PROXY protocol. Get this wrong and your IP allowlist will see the load balancer IP instead of the actual client.

flowchart LR
    subgraph "Direct (NLB)"
        C1[Client<br/>203.0.113.10] --> |"PROXY protocol"| NLB1[NLB]
        NLB1 --> |"Client IP in<br/>PROXY header"| E1[Envoy]
    end

    subgraph "Behind CDN"
        C2[Client<br/>203.0.113.10] --> CDN[CDN]
        CDN --> |"X-Forwarded-For:<br/>203.0.113.10"| NLB2[NLB]
        NLB2 --> E2[Envoy<br/>trustedHops: 1]
    end

    style NLB1 fill:#4ecdc4,color:#000
    style CDN fill:#ffd93d,color:#000

OAuth2 Proxy: If You Need Auth

This is where the Gateway API really shines over ingress-nginx. With ingress-nginx, adding OAuth2 authentication meant:

1. Deploy oauth2-proxy as a separate Deployment + Service
2. Add auth-url and auth-signin annotations to every Ingress
3. Hope the cookie domains line up
4. Debug redirect loops

flowchart LR
    subgraph "ingress-nginx OAuth2 (painful)"
        C[Client] --> NGINX2[ingress-nginx]
        NGINX2 --> |"auth-url annotation"| OAUTH[oauth2-proxy<br/>Deployment]
        OAUTH --> |"redirect"| IDP1[Identity Provider]
        NGINX2 --> |"if authed"| APP1[App]
    end

    style OAUTH fill:#ff6b6b,color:#fff

Envoy Gateway supports OIDC natively through SecurityPolicy - no separate proxy deployment needed:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-auth
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: protected-app
  oidc:
    provider:
      issuer: https://accounts.google.com
    clientID: your-client-id
    clientSecret:
      name: oidc-client-secret
    redirectURL: https://app.example.com/oauth2/callback
    scopes:
      - openid
      - email
      - profile

flowchart LR
    subgraph "Envoy Gateway OIDC (native)"
        C2[Client] --> ENVOY2[Envoy Proxy]
        ENVOY2 --> |"built-in OIDC"| IDP2[Identity Provider]
        IDP2 --> |"token"| ENVOY2
        ENVOY2 --> |"if authed"| APP2[App]
    end

    style ENVOY2 fill:#4ecdc4,color:#000

No extra deployment. No annotation wiring. The Envoy proxy handles the OIDC flow directly. You can even combine it with IP allowlisting - whitelist your office IPs and require OAuth2 for everyone else.

For simpler use cases, there’s also JWT validation and API key auth:

# JWT validation (for API endpoints)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
spec:
  targetRefs:
    - kind: HTTPRoute
      name: api-routes
  jwt:
    providers:
      - name: auth-service
        issuer: https://auth.example.com
        audiences:
          - api.example.com
        remoteJWKS:
          uri: https://auth.example.com/.well-known/jwks.json

Cross-Namespace Routing

One thing I had to solve: the Gateway lives in its own namespace, but application HTTPRoutes live in application namespaces. Kubernetes won’t allow cross-namespace references by default.

ReferenceGrants fix this:

# templates/reference-grants.yaml
{{- range .Values.gateway.listeners }}
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-routes-{{ .name }}
  namespace: {{ $.Release.Namespace }}
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: {{ .namespace }}
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
      namespace: {{ .namespace }}
  to:
    - group: gateway.networking.k8s.io
      kind: Gateway
{{- end }}

The chart auto-generates one per listener. Application teams don’t need to think about it.

flowchart TB
    subgraph "gateway namespace"
        GW[Gateway] --- RG1[ReferenceGrant<br/>allows: app-a ns]
        GW --- RG2[ReferenceGrant<br/>allows: app-b ns]
    end

    subgraph "app-a namespace"
        HR1[HTTPRoute: app-a] --> |"attaches to"| GW
    end

    subgraph "app-b namespace"
        HR2[HTTPRoute: app-b] --> |"attaches to"| GW
    end

    style GW fill:#4ecdc4,color:#000
    style RG1 fill:#ffd93d,color:#000
    style RG2 fill:#ffd93d,color:#000

The Data Plane: EnvoyProxy

The EnvoyProxy resource configures the actual Envoy instances that handle traffic. This is where your NLB annotations go:

# templates/envoy-proxy.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: {{ .Values.envoyProxy.name }}
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: {{ .Values.envoyProxy.service.type }}
        externalTrafficPolicy: {{ .Values.envoyProxy.service.externalTrafficPolicy }}
        annotations:
          # NLB configuration (same annotations you used with ingress-nginx)
          {{- toYaml .Values.envoyProxy.service.annotations | nindent 10 }}
      envoyDeployment:
        replicas: {{ .Values.envoyProxy.replicas }}

Values for an AWS NLB setup:

envoyProxy:
  name: envoy-proxy-config
  replicas: 2
  service:
    type: LoadBalancer
    externalTrafficPolicy: Local
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-name: my-gateway
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

The NLB annotations are the same ones you used with ingress-nginx. The difference is they’re on the EnvoyProxy resource instead of the ingress-nginx Service.

The Full Request Flow

From client to pod, here’s what a request looks like:

flowchart LR
    CLIENT[Client] --> NLB[NLB<br/>TLS passthrough]
    NLB --> |"PROXY protocol v2"| ENVOY[Envoy Proxy<br/>TLS termination]

    ENVOY --> |"1. ClientTrafficPolicy<br/>Extract client IP"| CTP[Client IP<br/>detected]
    CTP --> |"2. SecurityPolicy<br/>Check IP allowlist"| SP{IP Allowed?}
    SP --> |"No"| DENY[403 Forbidden]
    SP --> |"Yes"| BTP{Rate limit<br/>exceeded?}
    BTP --> |"Yes"| LIMIT[429 Too Many<br/>Requests]
    BTP --> |"No"| ROUTE[HTTPRoute<br/>matches host + path]
    ROUTE --> POD[Backend Pod]

    style ENVOY fill:#4ecdc4,color:#000
    style DENY fill:#ff6b6b,color:#fff
    style LIMIT fill:#ff6b6b,color:#fff
    style POD fill:#96ceb4,color:#000

The policies execute in order: client IP detection first, then security (IP allowlist, auth), then traffic policies (rate limiting), then routing to the backend.

Migration Strategy

We didn’t do a big-bang migration. The approach:

flowchart TD
    subgraph "Phase 1: Deploy Side-by-Side"
        NGINX[ingress-nginx<br/>existing traffic] --> APPS1[All Apps]
        EG[Envoy Gateway<br/>staging only] --> APPS2[Staging App]
    end

    subgraph "Phase 2: Move Non-Critical"
        NGINX2[ingress-nginx<br/>critical apps] --> APPS3[Prod Apps]
        EG2[Envoy Gateway<br/>staging + internal] --> APPS4[Staging + Internal]
    end

    subgraph "Phase 3: Full Migration"
        EG3[Envoy Gateway<br/>all traffic] --> APPS5[All Apps]
        NGINX3[ingress-nginx<br/>decommissioned] ~~~ GONE[Removed]
    end

    style NGINX fill:#ff6b6b,color:#fff
    style NGINX2 fill:#ff6b6b,color:#fff
    style NGINX3 fill:#ff6b6b,color:#fff
    style EG fill:#4ecdc4,color:#000
    style EG2 fill:#4ecdc4,color:#000
    style EG3 fill:#4ecdc4,color:#000
    style GONE fill:#e8e8e8,color:#666

1. Deploy Envoy Gateway alongside ingress-nginx on a separate NLB
2. Migrate staging first - prove the Gateway, policies, and certs work
3. Move non-critical production routes - internal tools, monitoring dashboards
4. Migrate remaining production traffic - switch DNS, verify, decommission ingress-nginx

The Helm chart supports this: set gateway.enabled: false in production while staging runs the full stack. Flip it to true when ready.

ingress-nginx vs Envoy Gateway

Feature	ingress-nginx	Envoy Gateway
API	Ingress (v1) + annotations	Gateway API (v1) + policy CRDs
Rate limiting	Global annotation per Ingress	Per-route BackendTrafficPolicy
IP allowlisting	Annotation per Ingress	SecurityPolicy targeting routes
OAuth2/OIDC	Separate oauth2-proxy deployment	Native OIDC in SecurityPolicy
TLS certificates	cert-manager + Ingress annotations	cert-manager + Gateway listeners
Per-route policies	Not really (annotation inheritance issues)	First-class policy attachment
gRPC	Annotation-based	Native GRPCRoute resource
Status	EOL March 2026	Active development

---

Takeaways

1. The Gateway API is what Ingress should have been. Separate resources for routing, security, and traffic policies. No more annotation soup.

2. Policy attachment is the killer feature. Target a rate limit or IP allowlist to a specific HTTPRoute, not globally. This alone justified the migration.

3. OIDC without oauth2-proxy. Envoy Gateway handles the OIDC flow natively. One less deployment to manage, one less thing to break.

4. Migrate incrementally. Run both controllers side by side. Move staging first, production last. The Helm chart supports toggling per environment.

5. Client IP detection matters. Get your ClientTrafficPolicy right (PROXY protocol vs X-Forwarded-For) before relying on IP-based policies. Wrong config means your allowlist sees the load balancer IP.

6. Cross-namespace routing needs ReferenceGrants. The Gateway API is explicit about namespace boundaries. Automate the grants in your chart so app teams don’t have to think about it.

7. Your NLB annotations carry over. The AWS load balancer annotations are the same - they just move from the ingress-nginx Service to the EnvoyProxy resource.

The Gateway API isn’t just a replacement for Ingress - it’s a fundamentally better model for managing traffic into your cluster. If you’re still on ingress-nginx, the EOL deadline is a good forcing function to make the switch.