Not all metrics are deleted after removing ingress object from kubernetes cluster

[dkhachyan]

What happened:

After creating ingress object added host metrics with status="404" and with empty ingress and namespace labels.
After removing this ingress object such metrics not being deleted

What you expected to happen:

Ingress controller ignore metric event from new host until nginx config reload complete.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v1.9.3

Kubernetes version (use kubectl version): v1.27.5

Environment:

• Cloud provider or hardware configuration: baremetal

• OS (e.g. from /etc/os-release): Ubuntu 20.04.5 LTS

• Kernel (e.g. uname -a): 5.15.0-43-generic

• How was the ingress-nginx-controller installed: Kubespray

How to reproduce this issue:

• Create a lot of ingress objects(2000+)
• Create one more ingress object
• Perform requests to new ingress host
• Get status 404(nginx config not ready yet)
• Wait for status 200(nginx reload complete)
• Scrape controller metrics and check that metrics with status="404" label don`t have empty ingress and namespace labels
• Remove ingress object
• Scrape controller metrics
• Check that metrics with status="404" label and empty namespace and ingress labels haven`t been removed

Anything else we need to know:

In very large environments during nginx config reload process ingress controller exposes metrics with status="404" label and
empty namespace="" and ingress="" labels what causes that such metrics cat not be deleted.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

the timeseries data is retained but new data is not appended. so does not seem like a bug. wait for other comments.

/remove-kind bug

[dkhachyan]

Yes the new data is not appended but this metric will remain after deleting ingress object because of empty ingress and namespace labels

ingress-nginx/internal/ingress/metric/collectors/socket.go

Line 501 in 1bc7456

ns, ok := labels["namespace"]

I think the problem is here

ingress-nginx/internal/ingress/controller/controller.go

Line 187 in 1bc7456

n.metricCollector.SetHosts(hosts)

host list was updated before the new host was added to the nginx config

May be I should add something like this

ingress-nginx/internal/ingress/metric/collectors/socket.go

Line 327 in 1bc7456

if sc.metricsPerHost && !sc.hosts.Has(stats.Host) {

if sc.metricsPerHost {
	if !sc.hosts.Has(stats.Host) {
		klog.V(3).InfoS("Skipping metric for host not being served", "host", stats.Host)
		continue
	}
	if stats.Host != "" {
		if stats.Namespace == "" || stats.Ingress == "" {
			continue
		}
}

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.