Building a Trace => Meter firehose

[mcrookbunnings]

Hey All!

I am utilising metrics for some pretty useful things, as such as getting a count of failed vs successful requests over given time spans and I wanted to improve this to also record metrics of successful/failed Azure Cosmos requests.

Now the Azure SDK does provide traces for consumption, however I cannot find a way to convert a stream of traces into metrics... I have created a BaseProcessor<Activity> POC which overrides OnEnd and dumps the source name, which does work HOWEVER I am unable to have both a sampled set of data sent off to my Otel backing services AND also have my Processor receive all incoming Traces.

tracingOptions.AddProviderConfiguration(builder => builder
                        .AddProcessor(new TracerProcesser())

Is there some way to take all the incoming Traces and pipe them through some form of processor/hose/middleware that would allow me to push out metrics?

Note: Very long day, will attempt to re-word this OR clarify things in the morning

[cijothomas]

You may find it easier to use OTelCollector to build a Span->Metrics pipeline (assuming you can use Collector).

If trying to do this in-proc completely, you need to write custom sampler, to ensure Activity is always created (so that the processor can see it, and generate metrics out of it), but not exported (so as to keep your costs in control).

AzureMonitor is doing the above if you want to take a look.( Warning: it is very custom to Azure, so you'll have to do a lot of customizations, but the idea is borrowable)
/~https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/StandardMetricsExtractionProcessor.cs

[mcrookbunnings]

amazing answer and so quick to boot! I will start investigating

[mcrookbunnings]

I recreated a simple Processor that logs "Foo" each time it is invoked as a way to test if Sampling is being applied as shown:

using System;
using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;

namespace FooBarIndustries.Extensions
{
    public static class CosmosdbExporterExtensions
    {
        public static TracerProviderBuilder AddCosmosdbMetricExporter(this TracerProviderBuilder builder)
        {
            if (builder == null)
            {
                throw new ArgumentNullException(nameof(builder));
            }

            var deferredBuilder = builder as IDeferredTracerProviderBuilder;
            if (deferredBuilder == null)
            {
                throw new InvalidOperationException(
                    "The provided TracerProviderBuilder does not implement IDeferredTracerProviderBuilder.");
            }

            return deferredBuilder.Configure((sp, builder) =>
            {
                builder.SetSampler(new AlwaysOnSampler());

                builder.AddProcessor(new StandardMetricsExtractionProcessor());
            });
        }
    }

    internal sealed class StandardMetricsExtractionProcessor : BaseProcessor<Activity>
    {
        public override void OnEnd(Activity activity)
        {
            Console.WriteLine("Foo");
        }
    }
}

However when I hook it into my existing telemetry setup (in which I use a sampling rate of 0.1 as we couldn't upload every trace) and it still ends up using whatever the latest registered sampler is NOT the AlwaysOnSampler as configured inside of deferredBuilder.Configure this leads to not every trace executing inside of OnEnd.

I would post the Otel config we have however it's not really viable for me to do so, but basically we call hostBuilder.ConfigureLogging from there allot of setup occurs, but it ends up boiling down to a call to serviceCollection.AddOpenTelemetry().WithTracing in which a list of TracerProviderBuilder's are passed in from a prior configuration, one of those is my AddCosmosdbMetricExporter which is setup via:

builder => builder.AddCosmosdbMetricExporter()

Long story short, there is a rather complicated beast behind our configuration shared by many projects at the org which makes the odds that it's just something funky happening on my end VERY likey. With all that said I thought I would roll the die and hope maybe something stuck out that you think I could trial.