Add exemplar support

README.md

@@ -1,8 +1,11 @@
 ![GitHub_headerImage](https://user-images.githubusercontent.com/3262610/221191767-73b8a8d9-9f8b-440e-8ab6-75cb3c82f2bc.png)
 
-# autometrics-py
+[![Discord Shield](https://discordapp.com/api/guilds/950489382626951178/widget.png?style=shield)](https://discord.gg/kHtwcH8As9)
 
-A Python library that exports a decorator that makes it easy to understand the error rate, response time, and production usage of any function in your code. Jump straight from your IDE to live Prometheus charts for each HTTP/RPC handler, database method, or other piece of application logic.
+> A Python port of the Rust
+> [autometrics-rs](/~https://github.com/fiberplane/autometrics-rs) library
+
+**Autometrics is a library that exports a decorator that makes it easy to understand the error rate, response time, and production usage of any function in your code.** Jump straight from your IDE to live Prometheus charts for each HTTP/RPC handler, database method, or other piece of application logic.
 
 Autometrics for Python provides:
 
@@ -22,6 +25,7 @@ See [Why Autometrics?](/~https://github.com/autometrics-dev#why-autometrics) for m
 - [🚨 Define alerts](#alerts--slos) using SLO best practices directly in your source code
 - [📊 Grafana dashboards](#dashboards) work out of the box to visualize the performance of instrumented functions & SLOs
 - [⚙️ Configurable](#metrics-libraries) metric collection library (`opentelemetry`, `prometheus`, or `metrics`)
+- [📍 Attach exemplars](/~https://github.com/autometrics-dev/autometrics-py/blob/main/examples/starlette-otel-exemplars.py) to connect metrics with traces
 - ⚡ Minimal runtime overhead
 
 ## Using autometrics-py
@@ -130,6 +134,15 @@ The `version` is read from the `AUTOMETRICS_VERSION` environment variable, and t
 
 This follows the method outlined in [Exposing the software version to Prometheus](https://www.robustperception.io/exposing-the-software-version-to-prometheus/).
 
+## Exemplars
+
+> **NOTE** - As of writing, aren't supported by the default tracker (`AUTOMETRICS_TRACKER=OPEN_TELEMETRY`).
+> You can track the progress of this feature here: /~https://github.com/open-telemetry/opentelemetry-python/issues/2407
+>
+> Because this feature is currently experimental, you have to explicitly enable it by setting the environment variables `AUTOMETRICS_EXEMPLARS=true` and `AUTOMETRICS_TRACKER=prometheus`. Prometheus also doesn't support exemplars in default configuration, to enable them you need to launch Prometheus with the `--enable-feature=exemplar-storage` flag.
+
+Exemplars are a way to attach additional information to a metric. If you have OpenTelemetry instrumentation set up, autometrics will try to attach the trace ID and span ID to the metrics. This allows you to jump from a metric to the trace that caused it.
+
 ## Development of the package
 
 This package uses [poetry](https://python-poetry.org) as a package manager, with all dependencies separated into three groups:

examples/README.md

@@ -5,6 +5,7 @@ You should be able to run each example by:
 - cloning this repository
 - run `poetry install --with examples`
 - and execute `poetry run python examples/<example>.py` from the root of the repo.
+- for django and starlette examples, you can find the instructions in their respective READMEs
 
 You can change the base url for Prometheus links via the `PROMETHEUS_URL` environment variable. So, if your local Prometheus were on a non-default port, like 9091, you would run:
 
@@ -57,3 +58,17 @@ Autometrics also tracks a label, `caller`, which is the name of the function tha
 This is an example that shows you how to use autometrics to get metrics on http handlers with FastAPI. In this case, we're setting up the API ourselves, which means we need to expose a `/metrics` endpoint manually.
 
 > Don't forget to configure Prometheus itself to scrape the metrics endpoint. Refer to the example `prometheus.yaml` file in the root of this project on how to set this up.
+
+## `django-example`
+
+This is a default Django project with autometrics configured. You can find examples of instrumenting function and class based views in `django_example/views`.
+
+> Don't forget to configure Prometheus itself to scrape the metrics endpoint. Refer to the example `prometheus.yaml` file in the root of this project on how to set this up.
+
+## `starlette-otel-exemplars.py`
+
+This app shows how to use the OpenTelemetry integration to add exemplars to your metrics. In a distributed system, it allows you to track a request as it flows through your system by adding trace/span ids to it. We can catch these ids and expose them to Prometheus as exemplars. Do note that exemplars are an experimental feature and you need to enable it in Prometheus with a `--enable-feature=exemplar-storage` flag. Run the example with a command:
+
+`AUTOMETRICS_TRACKER=prometheus AUTOMETRICS_EXEMPLARS=true uvicorn starlette-otel-exemplars:app --port 3000`
+
+> Don't forget to configure Prometheus itself to scrape the metrics endpoint. Refer to the example `prometheus.yaml` file in the root of this project on how to set this up.

examples/starlette-otel-exemplars.py

@@ -0,0 +1,45 @@
+from opentelemetry import trace
+from autometrics import autometrics
+from prometheus_client import REGISTRY
+from prometheus_client.openmetrics.exposition import generate_latest
+from starlette import applications
+from starlette.responses import PlainTextResponse
+from starlette.routing import Route
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import (
+    BatchSpanProcessor,
+    ConsoleSpanExporter,
+)
+
+# Let's start by setting up the OpenTelemetry SDK with some defaults
+provider = TracerProvider()
+processor = BatchSpanProcessor(ConsoleSpanExporter())
+provider.add_span_processor(processor)
+trace.set_tracer_provider(provider)
+
+# Now we can instrument our Starlette application
+tracer = trace.get_tracer(__name__)
+
+
+# We need to add tracer decorator before autometrics so that we see the spans
+@tracer.start_as_current_span("hello")
+@autometrics
+def hello(request):
+    return PlainTextResponse("hello world")
+
+
+def metrics(request):
+    # Exemplars are not supported by default prometheus format, so we specifically
+    # make an endpoint that uses the OpenMetrics format that supoorts exemplars.
+    body = generate_latest(REGISTRY)
+    return PlainTextResponse(body, media_type="application/openmetrics-text")
+
+
+app = applications.Starlette(routes=[Route("/", hello), Route("/metrics", metrics)])
+
+# Now, start the app (env variables are required to enable exemplars):
+# AUTOMETRICS_TRACKER=prometheus AUTOMETRICS_EXEMPLARS=true uvicorn starlette-otel-exemplars:app --port 3000
+# And make some requests to /. You should see the spans in the console.
+# You can hover over the hello handler and see the links to the metrics provided by autometrics.
+# If you open the queries in prometheus, you should see span and trace id exemplars added to metrics.

examples/starlette.sh

@@ -0,0 +1 @@
+AUTOMETRICS_TRACKER=prometheus AUTOMETRICS_EXEMPLARS=true uvicorn starlette-otel-exemplars:app --port 3000