v1.12.0 seemingly breaks ingresses with auth snippets

[craigcabrey]

What happened:

Upgrade to v1.12.0 using Helm chart 4.12.0 (I do this using Argo if it matters). Many ingresses which make use of a forward auth seem to break with a 404.

What you expected to happen:

Existing ingresses continue to function as expected.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

before (working):

internal-ingress-nginx-controller-6d5597c8dc-2q2dv:/etc/nginx$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.4
  Build:         ba0f2ee37f032c9f11967b74862c60a43ed59b36
  Repository:    /~https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

after (broken):

internal-ingress-nginx-controller-6bd59b7898-zgrs4:/etc/nginx$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.0
  Build:         ba73b2c24d355f1cdcf4b31ef7c5574059f12118
  Repository:    /~https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Ingress with auth snippets applied (e.g. for Authentik integration):

  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/auth-response-headers: Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: https://auth.[snip]/outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: http://ak-outpost-authentik-embedded-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx

Kubernetes version (use kubectl version): v1.29.12+k3s1

Environment:

• Cloud provider or hardware configuration: Self hosted homelab

• OS (e.g. from /etc/os-release): Fedora CoreOS 41.20241109.3.0

• Kernel (e.g. uname -a): 6.11.6-300.fc41.x86_64

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed: Helm via Argo (not sure how to get the full release out of argo)

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:         external
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=external
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.11.3
              argocd.argoproj.io/instance=external-ingress
              helm.sh/chart=ingress-nginx-4.11.3
Annotations:  <none>
Controller:   k8s.io/external-ingress-nginx
Events:       <none>


Name:         internal
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=internal
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0
              argocd.argoproj.io/instance=internal-ingress
              helm.sh/chart=ingress-nginx-4.12.0
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/internal-ingress-nginx
Events:       <none>

Logs show it sees the ingress:

I0110 05:00:54.802592       7 store.go:440] "Found valid IngressClass" ingress="teslamate/prod-teslamate" ingressclass="internal"
I0110 05:00:54.802724       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"teslamate", Name:"prod-teslamate", UID:"f3ecd5ad-fa14-4a8e-808b-ebd93978f530", APIVersion:"networking.k8s.io/v1", ResourceVersion:"145859890", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0110 05:00:54.802799       7 backend_ssl.go:67] "Adding secret to local store" name="teslamate/teslamate.[snip]-prod-tls"

Anything else we need to know:

Reliably reproducible. No difference in Helm chart versions, I isolated it down to the v1.12.0 image of the nginx controller causing the unexpected behavior.

Ingresses wihtout auth snippets function as expected.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[abhiverma001]

@craigcabrey I also faced the same issue For this issue now we need to set controller.config.annotations-risk-level: Critical in order to use snippets due to breaking changes in a minor version bump..

The "risk" list is here.. Annotations Risks - Ingress-Nginx Controller

[Socolin]

I got the same problem, invalid TLS certificate after upgrading

@craigcabrey I also faced the same issue For this issue now we need to set controller.config.annotations-risk-level: Critical in order to use snippets due to breaking changes in a minor version bump..

This fixed the problem, Thanks !

I was not expecting a breaking change on minor version bump too.

[craigcabrey]

thanks, can confirm this also works for me. I'll leave this open for the maintainers to decide what to do with it.

[brennerm]

I was also caught by this off guard. Was doing a bit of Helm chart maintenance and suddenly 90% of my ingresses respond with a 404. Not sure if a change like this should've landed in a minor release. Even the PR marks it as a breaking change.

[vmstan]

@abhiverma001 thank you for sharing that link, I have been banging my head all morning trying to figure out why upgrading to 1.12 kept provisioning fake certificates.

[vrijesh]

Is this bug planned to get fixed in the future releases or expected to have the risk level annotations done to consume the latest images.

[ba-encodify]

the controller.config.annotations-risk-level: Criticaldosnt solve it for us

[nwp-mai]

the controller.config.annotations-risk-level: Criticaldosnt solve it for us

I also needed to set controller.enableAnnotationValidations: false.

[aadarshjain-dev]

I am also facing the same 404 error after upgrading to v1.12.0. Tried above solutions but still no luck.
Anyone knows what to do, or am I missing something here?

[longwuyuan]

Commenting on a 404 log message requires data on what specific snippets are configured and if snippets are allowed and risk level set etc etc.

[ba-encodify]

the ingress calling the auth

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: https://ba.panda.example.com/panda/rest/checkAuth
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_clear_headers "cross-origin-embedder-policy";
      more_clear_headers "cross-origin-resource-policy";
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: Keep-Alive,User-Agent,Access-Control-Allow-Origin,X-Datadog-Origin,X-Datadog-Parent-Id,X-Datadog-Sampling-Priority,X-Datadog-Trace-Id
    nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: https://ba.panda.example.com
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 8k
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
  labels:
    instance: panda-ba
  name: html-proxy-ba-panda-example-com
spec:
  ingressClassName: nginx
  rules:
  - host: proxy1.ba.panda.example.com
    http:
      paths:
      - backend:
          service:
            name: html-proxy
            port:
              number: 3100
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - proxy1.ba.panda.example.com
    secretName: ba-panda-example-com-private-tls

[ba-encodify]

The ingress serving the auth

kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/app-root: /panda/
    nginx.ingress.kubernetes.io/client-body-buffer-size: 1000000m
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/limit-connections: "200"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-cookie-path: ~^(.+)$ "/; domain=$host"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
  labels:
    system: panda-ba
  name: panda-ba-webapp-ingress
spec:
  rules:
  - host: ba.panda.example.com
    http:
      paths:
      - backend:
          service:
            name: panda-ba
            port:
              number: 8080
        path: /panda
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - ba.panda.example.com
    secretName: ba-panda-example-com-private-tls