Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metrics filter is not working #37124

Open
oszlak opened this issue Jan 9, 2025 · 4 comments
Open

metrics filter is not working #37124

oszlak opened this issue Jan 9, 2025 · 4 comments
Labels
processor/filter Filter processor waiting for author

Comments

@oszlak
Copy link

oszlak commented Jan 9, 2025

Component(s)

processor/filter

What happened?

Description

I'm trying to exclude metrics by metric name and attribute but it seems that the filter is ignored

Steps to Reproduce

creating a simple histogram metric and filtering it by some label

payment_duration = meter.create_histogram(
    "payment_processing_duration",
    description="Time spent processing payments",
    unit="ms"
)

def main():
    request_counter.add(1, {"currency": "EUR"})
    tracer = trace.get_tracer(__name__)
    payment_duration.record(150, {"status": "success", "payment_type": "credit"})
    payment_duration.record(300, {"status": "failed", "payment_type": "debit"})
    payment_duration.record(150, {"status": "123"})
    payment_duration.record(300, {"status": "125"})
    payment_duration.record(300, {"status": "1010"})
    payment_duration.record(300, {"status": "258", "payment_type": "debit"})


    requests.get("http://google.com")
    while(True):
        with tracer.start_as_current_span("main") as span:
            span.set_attribute("operation.value", 123)
            logger.info("Processing request")
            request_counter.add(1)

            with tracer.start_as_current_span("sub_operation") as sub_span:
                sub_span.set_attribute("sub.value", 321)
                logger.info("Processing sub-operation")
                request_counter.add(1)
        payment_duration.record(150, {"status": "success", "payment_type": "credit"})
        payment_duration.record(300, {"status": "failed", "payment_type": "debit"})
        payment_duration.record(150, {"status": "123"})
        payment_duration.record(300, {"status": "125"})
        payment_duration.record(300, {"status": "1010"})
        payment_duration.record(300, {"status": "258", "payment_type": "debit"})

and the filter:

processors:
 filter/exclude_metric:
   error_mode: ignore
   metrics:
     metric:
        - 'name == "payment_processing_duration" and resource.attributes["status"] == "123"'

Expected Result

the metric that match the name and label should be dropped

Actual Result

its not dropped

Collector version

0.94.0

Environment information

Environment

OS: macos 15
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
 otlp:
   protocols:
     http:

exporters:
 prometheus:
   endpoint: "0.0.0.0:8889"
   const_labels:
     label1: value1

 debug:
   verbosity: detailed

 zipkin:
   endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
   format: proto

 otlp:
   endpoint: jaeger-all-in-one:4317
   tls:
     insecure: true

processors:
 filter/exclude_metric:
   error_mode: ignore
   metrics:
     metric:
        - 'name == "payment_processing_duration" and resource.attributes["status"] == "123"'
 batch:

extensions:
 health_check:
 pprof:
   endpoint: :1888
 zpages:
   endpoint: :55679

service:
 extensions: [pprof, zpages, health_check]
 pipelines:
   traces:
     receivers: [otlp]
     processors: [batch]
     exporters: [debug, zipkin, otlp]
   metrics:
     receivers: [otlp]
     processors: [filter/exclude_metric, batch]
     exporters: [debug, prometheus]

Log output

otel-collector-1     | Metric #1
otel-collector-1     | Descriptor:
otel-collector-1     |      -> Name: payment_processing_duration
otel-collector-1     |      -> Description: Time spent processing payments
otel-collector-1     |      -> Unit: ms
otel-collector-1     |      -> DataType: Histogram
otel-collector-1     |      -> AggregationTemporality: Cumulative
otel-collector-1     | HistogramDataPoints #0
otel-collector-1     | Data point attributes:
otel-collector-1     |      -> status: Str(success)
otel-collector-1     |      -> payment_type: Str(credit)
otel-collector-1     | StartTimestamp: 2025-01-09 22:17:11.566159 +0000 UTC
otel-collector-1     | Timestamp: 2025-01-09 22:17:14.709457 +0000 UTC
otel-collector-1     | Count: 268
otel-collector-1     | Sum: 40200.000000
otel-collector-1     | Min: 150.000000
otel-collector-1     | Max: 150.000000
otel-collector-1     | ExplicitBounds #0: 0.000000
otel-collector-1     | ExplicitBounds #1: 5.000000
otel-collector-1     | ExplicitBounds #2: 10.000000
otel-collector-1     | ExplicitBounds #3: 25.000000
otel-collector-1     | ExplicitBounds #4: 50.000000
otel-collector-1     | ExplicitBounds #5: 75.000000
otel-collector-1     | ExplicitBounds #6: 100.000000
otel-collector-1     | ExplicitBounds #7: 250.000000
otel-collector-1     | ExplicitBounds #8: 500.000000
otel-collector-1     | ExplicitBounds #9: 750.000000
otel-collector-1     | ExplicitBounds #10: 1000.000000
otel-collector-1     | ExplicitBounds #11: 2500.000000
otel-collector-1     | ExplicitBounds #12: 5000.000000
otel-collector-1     | ExplicitBounds #13: 7500.000000
otel-collector-1     | ExplicitBounds #14: 10000.000000
otel-collector-1     | Buckets #0, Count: 0
otel-collector-1     | Buckets #1, Count: 0
otel-collector-1     | Buckets #2, Count: 0
otel-collector-1     | Buckets #3, Count: 0
otel-collector-1     | Buckets #4, Count: 0
otel-collector-1     | Buckets #5, Count: 0
otel-collector-1     | Buckets #6, Count: 0
otel-collector-1     | Buckets #7, Count: 268
otel-collector-1     | Buckets #8, Count: 0
otel-collector-1     | Buckets #9, Count: 0
otel-collector-1     | Buckets #10, Count: 0
otel-collector-1     | Buckets #11, Count: 0
otel-collector-1     | Buckets #12, Count: 0
otel-collector-1     | Buckets #13, Count: 0
otel-collector-1     | Buckets #14, Count: 0
otel-collector-1     | Buckets #15, Count: 0
otel-collector-1     | HistogramDataPoints #1
otel-collector-1     | Data point attributes:
otel-collector-1     |      -> status: Str(failed)
otel-collector-1     |      -> payment_type: Str(debit)
otel-collector-1     | StartTimestamp: 2025-01-09 22:17:11.566159 +0000 UTC
otel-collector-1     | Timestamp: 2025-01-09 22:17:14.709457 +0000 UTC
otel-collector-1     | Count: 268
otel-collector-1     | Sum: 80400.000000
otel-collector-1     | Min: 300.000000
otel-collector-1     | Max: 300.000000
otel-collector-1     | ExplicitBounds #0: 0.000000
otel-collector-1     | ExplicitBounds #1: 5.000000
otel-collector-1     | ExplicitBounds #2: 10.000000
otel-collector-1     | ExplicitBounds #3: 25.000000
otel-collector-1     | ExplicitBounds #4: 50.000000
otel-collector-1     | ExplicitBounds #5: 75.000000
otel-collector-1     | ExplicitBounds #6: 100.000000
otel-collector-1     | ExplicitBounds #7: 250.000000
otel-collector-1     | ExplicitBounds #8: 500.000000
otel-collector-1     | ExplicitBounds #9: 750.000000
otel-collector-1     | ExplicitBounds #10: 1000.000000
otel-collector-1     | ExplicitBounds #11: 2500.000000
otel-collector-1     | ExplicitBounds #12: 5000.000000
otel-collector-1     | ExplicitBounds #13: 7500.000000
otel-collector-1     | ExplicitBounds #14: 10000.000000
otel-collector-1     | Buckets #0, Count: 0
otel-collector-1     | Buckets #1, Count: 0
otel-collector-1     | Buckets #2, Count: 0
otel-collector-1     | Buckets #3, Count: 0
otel-collector-1     | Buckets #4, Count: 0
otel-collector-1     | Buckets #5, Count: 0
otel-collector-1     | Buckets #6, Count: 0
otel-collector-1     | Buckets #7, Count: 0
otel-collector-1     | Buckets #8, Count: 268
otel-collector-1     | Buckets #9, Count: 0
otel-collector-1     | Buckets #10, Count: 0
otel-collector-1     | Buckets #11, Count: 0
otel-collector-1     | Buckets #12, Count: 0
otel-collector-1     | Buckets #13, Count: 0
otel-collector-1     | Buckets #14, Count: 0
otel-collector-1     | Buckets #15, Count: 0
otel-collector-1     | HistogramDataPoints #2
otel-collector-1     | Data point attributes:
otel-collector-1     |      -> status: Str(123)

Additional context

No response
@oszlak oszlak added bug Something isn't working needs triage New item requiring triage labels Jan 9, 2025
@github-actions github-actions bot added the processor/filter Filter processor label Jan 9, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@oszlak
Copy link
Author

oszlak commented Jan 9, 2025

image

@JaredTan95
Copy link
Member

you can try run like this to check out if payment_processing_duration has been dropped:

processors:
 filter/exclude_metric:
   error_mode: ignore
   metrics:
     metric:
        - IsMatch(name, "payment_processing_duration*" )

@github-actions github-actions bot mentioned this issue Jan 14, 2025
Closed
@TylerHelmuth
Copy link
Member

@oszlak try enabling collector debug logs as well, it will print out exactly what the collector sees in the filterprocessor.

service:
  telemetry:
    logs:
      level: debug

@TylerHelmuth TylerHelmuth added waiting for author and removed bug Something isn't working needs triage New item requiring triage labels Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
processor/filter Filter processor waiting for author
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@oszlak @TylerHelmuth @JaredTan95