[processor/k8sattributes] Allow to extract metadata from other namespaces

[sathieu]

Component(s)

processor/k8sattributes

Is your feature request related to a problem? Please describe.

We have a notion of tenant. The tenant name is a label attached to a namespace.

Like this:

      k8sattributes:
        extract:
          labels:
          # ...
          - from: namespace
            key: kubitus-project.gitlab.io/tenant  # << label name
            tag_name: kubitus.tenant.name  # << attribute
          metadata:
          - k8s.namespace.name
          # ...
        filter:
          node_from_env_var: K8S_NODE_NAME
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection

We use this to set tenant using a transform:

      transform/loki-tenant:
        error_mode: ignore
        log_statements:
        - context: log
          statements:
          # default tenant id
          - set(resource.attributes["tenant_id"], "1")
          # use namespace label when available
          - set(resource.attributes["tenant_id"], resource.attributes["kubitus.tenant.name"])
            where resource.attributes["kubitus.tenant.name"] != nil

But this is not enough because we have some logs where the tenant should be extracted from a namespace which is not the pod namespace. For example we have ingress-nginx logs in the ingress-nginx namespace where the ingress namespace is found.

Here is what we tried:

      transform/loki-tenant:
        error_mode: ignore
        log_statements:
        - context: log
          statements:
          # set namespace from pod namespace
          - set(cache["tenant_namespace"], resource.attributes["k8s.namespace.name"])
          # if pod namespace is ingress-nginx, extract namespace from JSON body
          - set(cache["ingress_nginx_body"], ParseJSON(body))
            where IsMatch(cache["tenant_namespace"], "^ingress-nginx") and IsMatch(body, "^\\{")
          - set(cache["tenant_namespace"], cache["ingress_nginx_body"]["namespace"])
            where cache["ingress_nginx_body"]["namespace"] != nil
          # default tenant id
          - set(resource.attributes["tenant_id"], "1")
          # use namespace label when available. BUT we want it from cache["tenant_namespace"]!
          - set(resource.attributes["tenant_id"], resource.attributes["kubitus.tenant.name"])
            where resource.attributes["kubitus.tenant.name"] != nil

Describe the solution you'd like

Currently k8sattributes does several things:

1. extract POD ID
2. set resource attributes based on above step
3. if passthru: stop
4. set resource attributes for pod
5. get namespace name
6. get attributes for pod namespace
7. set namespace attributes based on above step
8. get attributes for pod node
9. set nodeattributes based on above step

I want only step 6, extracting attributes for a namespace that is not necessarily a pod namespace.

Describe alternatives you've considered

Injecting into k8s.namespace.name and reverting after. This is hackish and I couln't get it to work.

Additional context

No response

[github-actions]

Pinging code owners:

• processor/k8sattributes: @dmitryax @fatsheep9146 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[braydonk]

Triage: Issue appears to be well specified and has necessary info. Moving to waiting-for-codeowners

[TylerHelmuth]

@sathieu I see what you're doing with the transform processor, but it isn't clear to me what you need to the k8sattributes processor to do. It is capable of extracting labels from namespaces, but currently only the namespace associated to the data.

[sathieu]

@TylerHelmuth I want to extract metadata associated with cache["tenant_namespace"] namespace which is different from resource.attributes["k8s.namespace.name"] (ingress-nginx). It's OK for me to store this in an attribute or resource attribute.

[TylerHelmuth]

Ok ya, the k8sattributes processor does not currently support defining the namespace to extract data from for a given pod.

At the moment it is able to associate that telemetry came from a pod came in namespace A, and then associate namespace A metadata with the telemetry. It cannot be told that telemetry coming from a pod in namespace A be associated with metadata from namespace B.