LoadBalancer targets too many nodes

[lloesche]

Hi there, I feel like I'm doing something very obvious wrong but I can't seem to figure out what it is and I'm unsure if I messed up the config or if the cloud controller is having some snafu.

I deployed an ingress-nginx as a Deployment with 3 replicas. The LoadBalancer Service object is created with the following kubectl -n ingress-nginx get service/ingress-nginx-controller -o yaml output. Sorry for the long object, I didn't want to leave anything out, but the interesting sections are likely annotations and spec.selector:

apiVersion: v1
kind: Service
metadata:
  annotations:
    load-balancer.hetzner.cloud/hostname: lb.eu.fixcloud.io
    load-balancer.hetzner.cloud/http-redirect-https: "true"
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/name: fix-ingress
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  creationTimestamp: "2023-09-08T17:54:47Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.8.1
    helm.sh/chart: ingress-nginx-4.7.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "4928"
  uid: 196caba9-f30f-4adc-bbfe-3b3107322611
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.7.184
  clusterIPs:
  - 10.43.7.184
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 32675
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 31185
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: lb.eu.fixcloud.io

When I check their status everything looks good:

minimi:conf lukas$ kubectl -n ingress-nginx get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-6fcfdf46b9-927xc   1/1     Running   0          44m
pod/ingress-nginx-controller-6fcfdf46b9-bl6v8   1/1     Running   0          44m
pod/ingress-nginx-controller-6fcfdf46b9-czclp   1/1     Running   0          44m

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP         PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.43.138.156   lb.eu.fixcloud.io   80:30825/TCP,443:32557/TCP   44m
service/ingress-nginx-controller-admission   ClusterIP      10.43.216.4     <none>              443/TCP                      44m
service/ingress-nginx-controller-metrics     ClusterIP      10.43.56.170    <none>              10254/TCP                    44m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   3/3     3            3           44m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-6fcfdf46b9   3         3         3       44m

And if I query that selector from the LoadBalancer spec above it returns exactly the pods I would expect it to return:

minimi:conf lukas$ kubectl -n ingress-nginx get pods -l app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6fcfdf46b9-927xc   1/1     Running   0          45m
ingress-nginx-controller-6fcfdf46b9-bl6v8   1/1     Running   0          45m
ingress-nginx-controller-6fcfdf46b9-czclp   1/1     Running   0          45m

However, when I check the created load balancer it targets all the nodes of the K8s cluster, not just the three where those nginx pods are running:

So the setup kinda works, but this will become a problem once the cluster has more than the 25 allowed nodes.

When I check the hcloud-cloud-controller-manager logs I see that for some reason it creates the LB with all the nodes of the cluster:

I0908 17:54:47.486545       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0908 17:54:47.513037       1 load_balancers.go:109] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-controller" nodes=["fixsaas-cpx51-pool-workers-worker1","fixsaas-cpx51-pool-workers-worker2","fixsaas-cpx51-pool-workers-worker3","fixsaas-cpx51-pool-workers-worker4","fixsaas-cx21-master1","fixsaas-cx21-master3","fixsaas-cpx51-pool-db-worker1","fixsaas-cpx51-pool-db-worker2","fixsaas-cpx51-pool-db-worker3","fixsaas-cpx51-pool-workers-worker5","fixsaas-cx21-master2"]

I feel like I'm doing something very obvious wrong. What do I have to do so that the LoadBalancer only targets the nodes of the pods that the Service selector returns?

[xoxys]

This was reported multiple times already e.g. #373, as far as I can see no solution yet. There were also PRs to address the proposed solution #246, but without further response...

@apricote Would you still accept a PR to implement an LB node selector annotation?

[apricote]

For sure 👍 Sounds like a good thing to have, especially for larger clusters. I think the general guidance I gave in #373 (comment) still applies, but feel free to open a PR and ping me for additional questions.

[xoxys]

Awesome. Thanks!

[lloesche]

I think the general guidance I gave in #373 (comment) still applies

@apricote quick question about that. There you wrote:

We add a new annotation:
LBNodeSelector Name = "load-balancer.hetzner.cloud/node-selector"
This annotation contains a label selector string matching the string based format used in kubectl.

Why not use the selector that already comes with the LoadBalancer Service object?

  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx

Are there situations where this selector leads to incorrect backends?

[apricote]

The Service Selector selects the Pods that the service targets.

The new annotation will select the Nodes that we add as targets for the Load Balancer.

---

When you create a Service type: LoadBalancer, Kubernetes assigns NodePorts for every Port you have configured in the Service. These Node Ports are opened on every Node of the Kubernetes cluster, and accept traffic for the Service. Internally, when traffic arrives at one the Node Ports, the requests are redirected to one of the Pods that belong to the Service. This also happens when the traffic arrives on a Node that does not have any of the expected Pods.

k8s.io/cloud-provider (which we use internally) hands us all Nodes to assign to the Load Balancer, and we add them to the Load Balancer as Targets. The traffic is then sent from the LB to the Node Ports of the service.

[lloesche]

The Service Selector selects the Pods that the service targets.

Right, my thinking was to have the operator resolve the nodes those pods run on and use them and the node ports as LB backends. That way ingress traffic would only end up on nodes that have ingress pods currently scheduled on them.

But given the freedom people have in their K8s networking setups I guess this makes too many assumptions, and having a dedicated label for the operator provides more flexibility.

[apricote]

Right, my thinking was to have the operator resolve the nodes those pods run on and use them and the node ports as LB backends. That way ingress traffic would only end up on nodes that have ingress pods currently scheduled on them.

That is outside of the scope of HCCM. We only implement the interfaces that k8s.io/cloud-provider requires, and do not talk to Kubernetes ourselves.