[dogstatsd] Make dogstatsd work better with containers

[mikekap]

Similar to how service discovery lets you probe metrics (pull), it would be awesome if some integration existed for statsd packets (push). This would likely take the form of adding tags to incoming metrics, similar to host.

Particularly, it would be great if e.g. for kubernetes, the pod name & deployment name were appended as tags. This information can be extracted from the source ip of the message (at least in Kubernetes).

[remh]

Thanks @mikekap this exact idea is already on our roadmap, it might take some time though as we'll need to consolidate a bunch of things for it to happen but we definitely agree on the use case.

[kiyose]

We are trying to work around this issue by setting up the kubernetes agent to run on each node via the documented daemonset configuration and also setting up an dogstatsd agent in each namespace, exposing 8125 via service. This runs afoul of at least one issue.

First, the agent that reports the stats may not be on the same host that the service that generated the stats is on. This means that we don't gain any benefit of automatically applied tags and it is harder to trace issues back to a given host.

Second, we have seen issues when we expose 8125 via a HostPort in the daemonset based agent with metrics being intermingled with name. This may be due to our network configuration and we are looking into it.

[danopia]

I'm running one dogstatsd per application container and want to tag each application's metrics with the app name. I tried setting tags: blah, blah, blah in the datadog.conf file but that doesn't seem to do anything. How can I universally tag all data send through one dogstatsd/forwarder pair?

That would relatively easily let me work around the noted issues here.

[pchristos]

What's the status of this?

[hkaj]

Hi @pchristos
This was implemented recently in agent 6 (not this repo, the new one at datadog-agent ), we're working on documentation right now but if you wanna see how it works here are 3 PRs that should help:

• server side: Dogstatsd: inject tagger tags via special entity tag datadog-agent#2969
• client side (go): Support configuring the client from standard environment variables datadog-go#78
• client side (python): add env vars support for dogstatsd host, port and entity id datadogpy#363

I'm going to close this issue as this won't be done in this repo (which is for the old agent 5). However if you face any issue with this feature do not hesitate to open an issue in datadog-agent.