4.12 breaks my application - possibly allowSnippetAnnotations: true?

[StefanLobbenmeierObjego]

What happened:

After upgrade, microservices no longer get requests, instead theyre forwarded to another ingress with wildcard

Requests are handles by this ingress apparently:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/acme-challenge-type: http01
    cert-manager.io/cluster-issuer: letsencrypt
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"cert-manager.io/acme-challenge-type":"http01","cert-manager.io/cluster-issuer":"letsencrypt","nginx.ingress.kubernetes.io/affinity-mode":"persistent","nginx.ingress.kubernetes.io/cors-allow-origin":"http://localhost:4200","nginx.ingress.kubernetes.io/enable-cors":"true","nginx.ingress.kubernetes.io/proxy-body-size":"15m","nginx.ingress.kubernetes.io/ssl-redirect":"true","nginx.ingress.kubernetes.io/use-forwarded-headers":"true","nginx.ingress.kubernetes.io/use-proxy-protocol":"true","nginx.ingress.kubernetes.io/use-regex":"true"},"creationTimestamp":"2020-09-21T18:56:52Z","generation":24,"name":"objego-ingress","namespace":"objego-int","resourceVersion":"393680605","uid":"622834ef-9ffe-4afc-86ca-1519a2875f11"},"spec":{"ingressClassName":"nginx","rules":[{"host":"app-int.objego.de","http":{"paths":[{"backend":{"service":{"name":"objego-document-storage-service","port":{"number":80}}},"path":"/documentstorage/api/","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-accounting","port":{"number":80}}},"path":"/accounting/","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-payment","port":{"number":80}}},"path":"/payment/api/","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-client","port":{"number":80}}},"path":"/","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-payment","port":{"number":80}}},"path":"/payment/public/api/billwerk/webhooks","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-payment","port":{"number":80}}},"path":"/payment/admin/api","pathType":"ImplementationSpecific"}]}}],"tls":[{"hosts":["app-int.objego.de"],"secretName":"letsencrypt-tls-secret"}]},"status":{"loadBalancer":{"ingress":[{"ip":"20.238.248.36"}]}}}
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/cors-allow-origin: http://localhost:4200
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 15m
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
    nginx.ingress.kubernetes.io/use-proxy-protocol: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
  creationTimestamp: "2020-09-21T18:56:52Z"
  generation: 28
  name: objego-ingress
  namespace: objego-int
  resourceVersion: "603596326"
  uid: 622834ef-9ffe-4afc-86ca-1519a2875f11
spec:
  ingressClassName: nginx
  rules:
  - host: app-int.objego.de
    http:
      paths:
      - backend:
          service:
            name: objego-client
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - app-int.objego.de
    secretName: letsencrypt-tls-secret
status:
  loadBalancer:
    ingress:
    - ip: 20.238.248.36

What you expected to happen:

requests to /api/userservice goes to ingress with /api/userservice, not to ingress with /

request should be handled by this ingress ( path: /userservice/api):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    actions.github.com/k8s-deploy: '{"run":"13267165487","repository":"objego/objego-user-service","workflow":"Deployment
      to Integration","workflowFileName":"k8s-deploy-failed-workflow-annotation","jobName":"deploy","createdBy":"renovate[bot]","runUri":"/~https://github.com/objego/objego-user-service/actions/runs/13267165487","commit":"c1c95ac2e1baf63b03ba2b0fcef14bdbedae3c18","lastSuccessRunCommit":"6c83a0787e164362caa88c451b231ca6d9f8429e","branch":"refs/heads/main","deployTimestamp":1739290277033,"dockerfilePaths":{},"manifestsPaths":["/~https://github.com/objego/objego-user-service/blob/c1c95ac2e1baf63b03ba2b0fcef14bdbedae3c18/deployment/shared","/~https://github.com/objego/objego-user-service/blob/c1c95ac2e1baf63b03ba2b0fcef14bdbedae3c18/deployment/int"],"helmChartPaths":[],"provider":"GitHub"}'
    azure-pipelines/jobName: '"Build Steps"'
    azure-pipelines/org: https://dev.azure.com/objego/
    azure-pipelines/pipeline: '"objego-user-service.git"'
    azure-pipelines/pipelineId: '"50"'
    azure-pipelines/project: objego.de
    azure-pipelines/run: "20230215.0"
    azure-pipelines/runuri: https://dev.azure.com/objego/objego.de/_build/results?buildId=22744
    cert-manager.io/acme-challenge-type: http01
    cert-manager.io/cluster-issuer: letsencrypt
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"cert-manager.io/acme-challenge-type":"http01","cert-manager.io/cluster-issuer":"letsencrypt","nginx.ingress.kubernetes.io/affinity-mode":"persistent","nginx.ingress.kubernetes.io/configuration-snippet":"# Omit the \"Origin\" header so CORS handling does not fall through to the application\nproxy_set_header Origin \"\";\n# Disable caching to prevent the client from storing confidential information in shared browser caches\nadd_header Cache-Control \"no-cache, no-store, must-revalidate\";\nadd_header Pragma \"no-cache\";\nadd_header Expires \"0\";\n","nginx.ingress.kubernetes.io/cors-allow-origin":"https://local.objego.de:4200","nginx.ingress.kubernetes.io/enable-cors":"true","nginx.ingress.kubernetes.io/proxy-body-size":"15m","nginx.ingress.kubernetes.io/ssl-redirect":"true","nginx.ingress.kubernetes.io/use-forwarded-headers":"true","nginx.ingress.kubernetes.io/use-regex":"true"},"name":"objego-user-service","namespace":"objego-int"},"spec":{"ingressClassName":"nginx","rules":[{"host":"app-int.objego.de","http":{"paths":[{"backend":{"service":{"name":"objego-user-service","port":{"number":80}}},"path":"/userservice/api","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-user-service","port":{"number":80}}},"path":"/userservice/auth","pathType":"ImplementationSpecific"},{"backend":{"service":{"name":"objego-user-service","port":{"number":80}}},"path":"/userservice/admin/api","pathType":"ImplementationSpecific"}]}}],"tls":[{"hosts":["app-int.objego.de"],"secretName":"letsencrypt-tls-secret-user-service"}]}}
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Omit the "Origin" header so CORS handling does not fall through to the application
      proxy_set_header Origin "";
      # Disable caching to prevent the client from storing confidential information in shared browser caches
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      add_header Pragma "no-cache";
      add_header Expires "0";
    nginx.ingress.kubernetes.io/cors-allow-origin: https://local.objego.de:4200
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 15m
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
  creationTimestamp: "2023-02-07T12:18:22Z"
  generation: 1
  labels:
    workflow: actions.github.com-k8s-deploy
    workflowFriendlyName: Deployment_to_Integration
  name: objego-user-service
  namespace: objego-int
  resourceVersion: "603596407"
  uid: 861ac876-2c29-4e5d-a009-afd3bf5d1a49
spec:
  ingressClassName: nginx
  rules:
  - host: app-int.objego.de
    http:
      paths:
      - backend:
          service:
            name: objego-user-service
            port:
              number: 80
        path: /userservice/api
        pathType: ImplementationSpecific
      - backend:
          service:
            name: objego-user-service
            port:
              number: 80
        path: /userservice/auth
        pathType: ImplementationSpecific
      - backend:
          service:
            name: objego-user-service
            port:
              number: 80
        path: /userservice/admin/api
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - app-int.objego.de
    secretName: letsencrypt-tls-secret-user-service
status:
  loadBalancer:
    ingress:
    - ip: 20.238.248.36

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Kubernetes version (use kubectl version):

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.29.11
WARNING: version difference between client (1.32) and server (1.29) exceeds the supported minor version skew of +/-1

Environment:

• Cloud provider or hardware configuration: Microsoft Azure AKS
• OS (e.g. from /etc/os-release): Ubuntu 22.04.5 LTS 5.15.0-1075-azure
• Kernel (e.g. uname -a): Linux nginx-ingress-nginx-controller-6fcd4674d9-kbpnh 5.15.0-1075-azure Detailed NGINX-Ingress-Controller status page config #84-Ubuntu SMP Mon Oct 21 15:42:52 UTC 2024 x86_64 Linux
• Install tools:
  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc. azure ui
• Basic cluster related info:
  • kubectl version

 Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.29.11
WARNING: version difference between client (1.32) and server (1.29) exceeds the supported minor version skew of +/-1

• kubectl get nodes -o wide

NAME                              STATUS   ROLES    AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-intmain-28895530-vmss000000   Ready    <none>   22d   v1.29.11   10.60.0.199   <none>        Ubuntu 22.04.5 LTS   5.15.0-1075-azure   containerd://1.7.23-1
aks-intmain-28895530-vmss000001   Ready    <none>   22d   v1.29.11   10.60.1.184   <none>        Ubuntu 22.04.5 LTS   5.15.0-1075-azure   containerd://1.7.23-1
aks-intmain-28895530-vmss000002   Ready    <none>   22d   v1.29.11   10.60.1.75    <none>        Ubuntu 22.04.5 LTS   5.15.0-1075-azure   containerd://1.7.23-1
aks-intmain-28895530-vmss000004   Ready    <none>   22d   v1.29.11   10.60.2.100   <none>        Ubuntu 22.04.5 LTS   5.15.0-1075-azure   containerd://1.7.23-1

• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress nginx ingress 50 2025-02-12 09:35:40.808695632 +0000 UTC deployed ingress-nginx-4.12.0 1.12.0
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  allowSnippetAnnotations: true
  config:
    limit-req-status-code: 429
  metrics:
    enabled: true
  podAnnotations:
    prometheus.io/port: 10254
    prometheus.io/scrape: true
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    externalTrafficPolicy: Local

• If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0
              helm.sh/chart=ingress-nginx-4.12.0
Annotations:  meta.helm.sh/release-name: nginx
              meta.helm.sh/release-namespace: ingress
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

-A is all namespaces, that is way too much output

kubectl -n ingress get all -o wide                                                                                   ⎈ AKS-Objego-Integration 10:38:26
NAME                                                  READY   STATUS        RESTARTS   AGE     IP           NODE                              NOMINATED NODE   READINESS GATES
pod/nginx-ingress-nginx-controller-6fcd4674d9-kbpnh   1/1     Running       0          3m25s   10.60.2.69   aks-intmain-28895530-vmss000004   <none>           <none>
pod/nginx-ingress-nginx-controller-bc595fb66-f2p7p    0/1     Terminating   0          31m     10.60.2.63   aks-intmain-28895530-vmss000004   <none>           <none>

NAME                                               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE      SELECTOR
service/nginx-ingress-nginx-controller             LoadBalancer   10.60.17.227   20.238.248.36   80:32493/TCP,443:30555/TCP   2y242d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-nginx-controller-admission   ClusterIP      10.60.17.108   <none>          443/TCP                      2y242d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
service/nginx-ingress-nginx-controller-metrics     ClusterIP      10.60.17.55    <none>          10254/TCP                    2y12d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE      CONTAINERS   IMAGES                                                                                                                     SELECTOR
deployment.apps/nginx-ingress-nginx-controller   1/1     1            1           2y242d   controller   registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                        DESIRED   CURRENT   READY   AGE    CONTAINERS   IMAGES                                                                                                                     SELECTOR
replicaset.apps/nginx-ingress-nginx-controller-54b76964f4   0         0         0       172d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=54b76964f4
replicaset.apps/nginx-ingress-nginx-controller-577b9bb7f8   0         0         0       110d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=577b9bb7f8
replicaset.apps/nginx-ingress-nginx-controller-5f94b54986   0         0         0       119d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5f94b54986
replicaset.apps/nginx-ingress-nginx-controller-64874ff947   0         0         0       121d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=64874ff947
replicaset.apps/nginx-ingress-nginx-controller-686db8c865   0         0         0       289d   controller   registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=686db8c865
replicaset.apps/nginx-ingress-nginx-controller-6b947865bb   0         0         0       207d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6b947865bb
replicaset.apps/nginx-ingress-nginx-controller-6c7cc4f9bf   0         0         0       110d   controller   registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c7cc4f9bf
replicaset.apps/nginx-ingress-nginx-controller-6c956c856b   0         0         0       354d   controller   registry.k8s.io/ingress-nginx/controller:v1.9.6@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c    app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c956c856b
replicaset.apps/nginx-ingress-nginx-controller-6fcd4674d9   1         1         1       50m    controller   registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6fcd4674d9
replicaset.apps/nginx-ingress-nginx-controller-bc595fb66    0         0         0       33m    controller   registry.k8s.io/ingress-nginx/controller:v1.11.4@sha256:981a97d78bee3109c0b149946c07989f8f1478a9265031d2d23dea839ba05b52   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=bc595fb66
replicaset.apps/nginx-ingress-nginx-controller-d8cd865ff    0         0         0       348d   controller   registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=d8cd865ff

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             nginx-ingress-nginx-controller-6fcd4674d9-kbpnh
Namespace:        ingress
Priority:         0
Service Account:  nginx-ingress-nginx
Node:             aks-intmain-28895530-vmss000004/10.60.2.100
Start Time:       Wed, 12 Feb 2025 10:35:51 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.12.0
                  helm.sh/chart=ingress-nginx-4.12.0
                  pod-template-hash=6fcd4674d9
Annotations:      kubectl.kubernetes.io/restartedAt: 2024-10-15T17:40:26+02:00
                  prometheus.io/port: 10254
                  prometheus.io/scrape: true
Status:           Running
IP:               10.60.2.69
IPs:
  IP:           10.60.2.69
Controlled By:  ReplicaSet/nginx-ingress-nginx-controller-6fcd4674d9
Containers:
  controller:
    Container ID:    containerd://4ddff8caf05af3d3e61db6ce56467503b7dd3018763d608ac6f187f405ad1b52
    Image:           registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa
    Ports:           80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --election-id=nginx-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-metrics=true
    State:          Running
      Started:      Wed, 12 Feb 2025 10:35:52 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-nginx-controller-6fcd4674d9-kbpnh (v1:metadata.name)
      POD_NAMESPACE:  ingress (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2fgfc (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-nginx-admission
    Optional:    false
  kube-api-access-2fgfc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From                      Message
  ----    ------     ----   ----                      -------
  Normal  Scheduled  4m12s  default-scheduler         Successfully assigned ingress/nginx-ingress-nginx-controller-6fcd4674d9-kbpnh to aks-intmain-28895530-vmss000004
  Normal  Pulled     4m12s  kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa" already present on machine
  Normal  Created    4m12s  kubelet                   Created container controller
  Normal  Started    4m12s  kubelet                   Started container controller
  Normal  RELOAD     4m9s   nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     nginx-ingress-nginx-controller
Namespace:                ingress
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.12.0
                          helm.sh/chart=ingress-nginx-4.12.0
Annotations:              meta.helm.sh/release-name: nginx
                          meta.helm.sh/release-namespace: ingress
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.60.17.227
IPs:                      10.60.17.227
LoadBalancer Ingress:     20.238.248.36
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32493/TCP
Endpoints:                10.60.2.69:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30555/TCP
Endpoints:                10.60.2.69:443
Session Affinity:         None
External Traffic Policy:  Local
Internal Traffic Policy:  Cluster
HealthCheck NodePort:     31434
Events:                   <none>

• Current state of ingress object, if applicable:
  • kubectl -n <appnamespace> get all,ing -o wide

all is again too much output, ing is manageable

NAME                                    CLASS   HOSTS                                    ADDRESS         PORTS     AGE
objego-accounting                       nginx   app-int.objego.de                                        80, 443   505d
objego-appendix-v                       nginx   app-int.objego.de                                        80, 443   3y25d
objego-banking                          nginx   app-int.objego.de                                        80, 443   2y269d
objego-calculation-management-service   nginx   app-int.objego.de                                        80, 443   2y237d
objego-content                          nginx   app-int.objego.de                                        80, 443   650d
objego-crm-adapter                      nginx   app-int.objego.de                                        80, 443   2y205d
objego-document-storage-service         nginx   app-int.objego.de                                        80, 443   452d
objego-elements                         nginx   app-int.objego.de                        20.238.248.36   80, 443   2y41d
objego-ingress                          nginx   app-int.objego.de                        20.238.248.36   80, 443   4y144d
objego-keycloak                         nginx   app-int.objego.de                                        80, 443   2y18d
objego-llama3                           nginx   llama.objego.io                                          80, 443   174d
objego-masterdata                       nginx   app-int.objego.de                                        80, 443   2y237d
objego-meter-management                 nginx   app-int.objego.de                                        80, 443   2y108d
objego-mixpanel-proxy                   nginx   app-int.objego.de                        20.238.248.36   80, 443   2y247d
objego-offering-engine                  nginx   app-int.objego.de,osw-int.objego.de                      80, 443   127d
objego-sales-wizard                     nginx   osw-int.objego.de                                        80, 443   203d
objego-strapi                           nginx   app-int.objego.de,strapi-int.objego.io   20.238.248.36   80, 443   2y270d
objego-subscription-adapter             nginx   app-int.objego.de                                        80, 443   2y179d
objego-user-service                     nginx   app-int.objego.de                                        80, 443   2y5d

• kubectl -n <appnamespace> describe ing <ingressname>

Name:             objego-user-service
Labels:           workflow=actions.github.com-k8s-deploy
                  workflowFriendlyName=Deployment_to_Integration
Namespace:        objego-int
Address:
Ingress Class:    nginx
Default backend:  <default>
TLS:
  letsencrypt-tls-secret-user-service terminates app-int.objego.de
Rules:
  Host               Path  Backends
  ----               ----  --------
  app-int.objego.de
                     /userservice/api         objego-user-service:80 (10.60.1.187:8091)
                     /userservice/auth        objego-user-service:80 (10.60.1.187:8091)
                     /userservice/admin/api   objego-user-service:80 (10.60.1.187:8091)
Annotations:         actions.github.com/k8s-deploy:
                       {"run":"13267165487","repository":"objego/objego-user-service","workflow":"Deployment to Integration","workflowFileName":"k8s-deploy-faile...
                     azure-pipelines/jobName: "Build Steps"
                     azure-pipelines/org: https://dev.azure.com/objego/
                     azure-pipelines/pipeline: "objego-user-service.git"
                     azure-pipelines/pipelineId: "50"
                     azure-pipelines/project: objego.de
                     azure-pipelines/run: 20230215.0
                     azure-pipelines/runuri: https://dev.azure.com/objego/objego.de/_build/results?buildId=22744
                     cert-manager.io/acme-challenge-type: http01
                     cert-manager.io/cluster-issuer: letsencrypt
                     nginx.ingress.kubernetes.io/affinity-mode: persistent
                     nginx.ingress.kubernetes.io/configuration-snippet:
                       # Omit the "Origin" header so CORS handling does not fall through to the application
                       proxy_set_header Origin "";
                       # Disable caching to prevent the client from storing confidential information in shared browser caches
                       add_header Cache-Control "no-cache, no-store, must-revalidate";
                       add_header Pragma "no-cache";
                       add_header Expires "0";
                     nginx.ingress.kubernetes.io/cors-allow-origin: https://local.objego.de:4200
                     nginx.ingress.kubernetes.io/enable-cors: true
                     nginx.ingress.kubernetes.io/proxy-body-size: 15m
                     nginx.ingress.kubernetes.io/ssl-redirect: true
                     nginx.ingress.kubernetes.io/use-forwarded-headers: true
                     nginx.ingress.kubernetes.io/use-regex: true
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    53m (x2 over 53m)     nginx-ingress-controller  Scheduled for sync
  Normal  Sync    40m (x2 over 45m)     nginx-ingress-controller  Scheduled for sync
  Normal  Sync    36m (x2 over 37m)     nginx-ingress-controller  Scheduled for sync
  Normal  Sync    35m                   nginx-ingress-controller  Scheduled for sync
  Normal  Sync    31m (x2 over 34m)     nginx-ingress-controller  Scheduled for sync
  Normal  Sync    6m38s (x2 over 7m1s)  nginx-ingress-controller  Scheduled for sync

• If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

curl 'https://app-int.objego.de/userservice/auth/public-key' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'priority: u=1, i' \
  -H 'referer: https://app-int.objego.de/login' \
  -H 'sec-ch-ua: "Not(A:Brand";v="99", "Brave";v="133", "Chromium";v="133"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36' \

Responds with HTML from objego-client instead of JSON from objego-user-service:

<!doctype html>
<html data-critters-container>
    <head lang="de">
        <meta charset="utf-8">
        <title>objego | einfach vermieten</title>
        <base href="/">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <link rel="icon" href="assets/favicon.png">
        <style>
            @keyframes spinning-loading-circle {
                to {
                    transform: rotate(359deg)
                }
            }

            body[data-loading]:before {
                content: "";
                width: 100vw;
                height: 100vh;
                background-color: #32323280;
                position: fixed;
                z-index: 10000;
                cursor: not-allowed
            }

            body[data-loading]:after {
                content: "";
                background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' height='82' width='82'%3E%3Ccircle cx='41' cy='41' r='30' fill='none' stroke='%23fcf488' stroke-width='8' stroke-dasharray='75' stroke-linecap='round' pathLength='100' /%3E%3C/svg%3E");
                background-position: center;
                background-repeat: no-repeat;
                width: 6.25rem;
                height: 6.25rem;
                position: fixed;
                animation: spinning-loading-circle 1s infinite linear;
                z-index: 10001;
                cursor: not-allowed
            }

            html {
                line-height: 1.15;
                -webkit-text-size-adjust: 100%
            }

            body {
                margin: 0
            }

            @font-face {
                font-family: AppFont;
                src: url(LatoLatin-Regular.9b027a406ebdbfd6.eot);
                src: url(LatoLatin-Regular.9b027a406ebdbfd6.eot?#iefix) format("embedded-opentype"),url(LatoLatin-Regular.d45d409d2fa42169.woff2) format("woff2"),url(LatoLatin-Regular.2d0c0323452c74d7.woff) format("woff"),url(LatoLatin-Regular.06c4cf9dfee1a8e3.ttf) format("truetype");
                font-style: normal;
                font-weight: 400;
                text-rendering:optimizeLegibility}

            @font-face {
                font-family: AppFont;
                src: url(LatoLatin-Bold.56af1572c88927d9.eot);
                src: url(LatoLatin-Bold.56af1572c88927d9.eot?#iefix) format("embedded-opentype"),url(LatoLatin-Bold.1cf13853095a62e2.woff2) format("woff2"),url(LatoLatin-Bold.9d1bee7824203b19.woff) format("woff"),url(LatoLatin-Bold.10d44a1a0ed5a99f.ttf) format("truetype");
                font-style: normal;
                font-weight: 700;
                text-rendering:optimizeLegibility}

            html,body {
                color: var(--color-text)
            }

            * {
                font-family: AppFont,Arial,sans-serif;
                box-sizing: border-box;
                -moz-osx-font-smoothing: grayscale
            }

            *,:before,:after {
                --tw-border-spacing-x: 0;
                --tw-border-spacing-y: 0;
                --tw-translate-x: 0;
                --tw-translate-y: 0;
                --tw-rotate: 0;
                --tw-skew-x: 0;
                --tw-skew-y: 0;
                --tw-scale-x: 1;
                --tw-scale-y: 1;
                --tw-pan-x: ;
                --tw-pan-y: ;
                --tw-pinch-zoom: ;
                --tw-scroll-snap-strictness: proximity;
                --tw-gradient-from-position: ;
                --tw-gradient-via-position: ;
                --tw-gradient-to-position: ;
                --tw-ordinal: ;
                --tw-slashed-zero: ;
                --tw-numeric-figure: ;
                --tw-numeric-spacing: ;
                --tw-numeric-fraction: ;
                --tw-ring-inset: ;
                --tw-ring-offset-width: 0px;
                --tw-ring-offset-color: #fff;
                --tw-ring-color: rgb(59 130 246 / .5);
                --tw-ring-offset-shadow: 0 0 #0000;
                --tw-ring-shadow: 0 0 #0000;
                --tw-shadow: 0 0 #0000;
                --tw-shadow-colored: 0 0 #0000;
                --tw-blur: ;
                --tw-brightness: ;
                --tw-contrast: ;
                --tw-grayscale: ;
                --tw-hue-rotate: ;
                --tw-invert: ;
                --tw-saturate: ;
                --tw-sepia: ;
                --tw-drop-shadow: ;
                --tw-backdrop-blur: ;
                --tw-backdrop-brightness: ;
                --tw-backdrop-contrast: ;
                --tw-backdrop-grayscale: ;
                --tw-backdrop-hue-rotate: ;
                --tw-backdrop-invert: ;
                --tw-backdrop-opacity: ;
                --tw-backdrop-saturate: ;
                --tw-backdrop-sepia: ;
                --tw-contain-size: ;
                --tw-contain-layout: ;
                --tw-contain-paint: ;
                --tw-contain-style:
            }

            :root {
                --color-primary: #41b9be;
                --color-primary-light: #e1f4f5;
                --color-primary-dark: #2f8689;
                --color-primary-default: #41b9be;
                --color-text: #0a1e1e;
                --color-secondary: #ffe100;
                --color-white: #ffffff;
                --color-bg-lite: #f8f8f8;
                --color-notification: #e1f4f5;
                --color-notification-badge: #f3595e;
                --color-label: #6c7878;
                --color-mandatory-field: #6c7878;
                --color-input-background: #f8f8f8;
                --color-danger: #f3595e;
                --color-success: #63c67c;
                --color-success-transparent: rgba(99, 198, 124, .1);
                --color-warning: #f5964c;
                --color-warning-transparent: rgba(245, 150, 76, .1);
                --color-danger-light: #fae4d6;
                --color-transparent: rgba(0, 0, 0, 0);
                --color-bg-fence: #ffe100;
                --color-bg-indicator: #ffec5c;
                --color-bg-fenced-surface: #e1f4f5;
                --color-bg-option-disabled: #e2e4e4;
                --color-gray: #d8dbdb;
                --color-muted: #9da5a5;
                --color-bg-light-gray: #ebeded;
                --color-annotation: #d8dbdb;
                --color-input-placeholder: #9da5a5;
                --color-input-border: #d8dbdb;
                --color-input-disabled: #6c7878;
                --color-actionbar-background: #ebeded;
                --color-onboarding: #3d589c;
                --color-dropdown-notallocatable: #ced2d2;
                --color-seperator: #ced2d2;
                --color-primary-hover: #3ba7ab;
                --color-primary-active: #215d5f;
                --color-primary-focus: #71cbce;
                --color-primary-focus-border: #2f8689;
                --color-primary-dark-hover: #215d5f;
                --color-primary-dark-active: #102e30;
                --color-primary-dark-focus: #41b9be;
                --color-primary-dark-focus-border: #e1f4f5;
                --color-secondary-hover: #e1f4f5;
                --color-secondary-active: #a0dcdf;
                --color-secondary-active-border: #215d5f;
                --color-secondary-active-text: #215d5f;
                --color-secondary-focus-border: #2f8689;
                --color-secondary-focus: #ecf8f9;
                --color-onboarding-default: #3d589c;
                --color-onboarding-hover: #354b86;
                --color-onboarding-active: #25355e;
                --color-onboarding-dark: #0c121f;
                --border-radius-default: .1875rem;
                --border-radius-textarea: .25rem;
                --border-size-default: .0625rem;
                --border-size-textarea: .12rem;
                --spacing-xxxs: .25rem;
                --spacing-xxs: .5rem;
                --spacing-xs: .625rem;
                --spacing-s: .75rem;
                --spacing-sm: 1rem;
                --spacing-md: 1.5rem;
                --spacing-lg: 2rem;
                --spacing-xl: 2.875rem;
                --spacing-xxl: 4rem;
                --spacing-default: 2rem;
                --font-size-paragraph-xxs: .5rem;
                --font-size-paragraph-xs: .625rem;
                --font-size-paragraph-sm: .875rem;
                --font-size-paragraph-md: 1rem;
                --font-size-subheading-1: 1.25rem;
                --font-size-subheading-2: 1.5rem;
                --font-size-headline: 1.75rem;
                --font-size-default: 1rem;
                --line-height-md: 1.375rem;
                --line-height-default: 1.375rem;
                --icon-size-default: 2rem;
                --icon-size-small: 1.125rem;
                --icon-size-medium: 3rem;
                --icon-size-sm: 1.5rem;
                --form-input-field-width: 17.5rem
            }

            html {
                line-height: 1.5;
                -webkit-text-size-adjust: 100%;
                font-family: Lato,ui-sans-serif,system-ui,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;
                -moz-tab-size: 4;
                tab-size: 4
            }

            *,:after,:before {
                --tw-border-spacing-x: 0;
                --tw-border-spacing-y: 0;
                --tw-translate-x: 0;
                --tw-translate-y: 0;
                --tw-rotate: 0;
                --tw-skew-x: 0;
                --tw-skew-y: 0;
                --tw-scale-x: 1;
                --tw-scale-y: 1;
                --tw-pan-x: ;
                --tw-pan-y: ;
                --tw-pinch-zoom: ;
                --tw-scroll-snap-strictness: proximity;
                --tw-gradient-from-position: ;
                --tw-gradient-via-position: ;
                --tw-gradient-to-position: ;
                --tw-ordinal: ;
                --tw-slashed-zero: ;
                --tw-numeric-figure: ;
                --tw-numeric-spacing: ;
                --tw-numeric-fraction: ;
                --tw-ring-inset: ;
                --tw-ring-offset-width: 0px;
                --tw-ring-offset-color: #fff;
                --tw-ring-color: rgba(59,130,246,.5);
                --tw-ring-offset-shadow: 0 0 #0000;
                --tw-ring-shadow: 0 0 #0000;
                --tw-shadow: 0 0 #0000;
                --tw-shadow-colored: 0 0 #0000;
                --tw-blur: ;
                --tw-brightness: ;
                --tw-contrast: ;
                --tw-grayscale: ;
                --tw-hue-rotate: ;
                --tw-invert: ;
                --tw-saturate: ;
                --tw-sepia: ;
                --tw-drop-shadow: ;
                --tw-backdrop-blur: ;
                --tw-backdrop-brightness: ;
                --tw-backdrop-contrast: ;
                --tw-backdrop-grayscale: ;
                --tw-backdrop-hue-rotate: ;
                --tw-backdrop-invert: ;
                --tw-backdrop-opacity: ;
                --tw-backdrop-saturate: ;
                --tw-backdrop-sepia: ;
                --tw-contain-size: ;
                --tw-contain-layout: ;
                --tw-contain-paint: ;
                --tw-contain-style:
            }

            body {
                justify-content: center;
                display: flex;
                align-items: center
            }

            app-root {
                height: 100vh;
                width: 100vw
            }
        </style>
        <link rel="stylesheet" href="styles.6b25d352f9bcb582.css" media="print" onload="this.media='all'">
        <noscript>
            <link rel="stylesheet" href="styles.6b25d352f9bcb582.css">
        </noscript>
    </head>
    <body data-loading>
        <app-root></app-root>
        <script src="runtime.eba6f091e49a47a6.js" type="module"></script>
        <script src="polyfills.818238552baeea70.js" type="module"></script>
        <script src="main.ec39d2b899baae57.js" type="module"></script>
    </body>
</html>

• Others:
  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[StefanLobbenmeierObjego]

will revert the upgrade now while filling out version to unblock my team / not keep the application broken for too long

[Gacko]

You need to increase the maximum allowed annotation risk level to Critical for snippet annotations: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#annotations-risk-level.

[StefanLobbenmeierObjego]

i see - can we also document this in https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations and add this as a breaking change to the release notes?

I guess this is marked with an ⚠ in the changelog but its hard to know the implications:

[Gacko]

There's documentation about annotations and their risk level: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations-risk/