Thanos Sidecar not able to reach

[pritster5]

I have Prometheus running via a HelmChart and I'm trying to add Thanos to it via sidecar containers.

I'm running into the following errors and warnings when flux recreates the prometheus and thanos containers:

level=error component=reloader msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 503 Service Unavailable; have you set `--web.enable-lifecycle` Prometheus flag?"

The flag in question is set under extraFlags below.

I'm also seeing these errors:

level=warn msg="changing probe status" status=not-ready reason="no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details."

and

level=warn msg="failed to get Prometheus flags. Is Prometheus running? Retrying" err="got non-200 response code: 503, response: no healthy upstream"

Is it possible that the external label issue is caused by the "No Healthy Upstream" issue?

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: prometheus
  namespace: redacted-ns
spec:
  chart:
    spec:
      chart: prometheus
      sourceRef:
        kind: GitRepository
        name: redacted-charts
        namespace: redacted-ns
      version: 25.20.1
  interval: 5m
  releaseName: prometheus
  serviceAccountName: flux2-redacted
  targetNamespace: monitoring
  values:
    serviceAccounts:
      server:
        create: true
        name: redacted-monitoring-prometheus
        annotations:
          eks.amazonaws.com/role-arn: redacted-arn
    configmapReload:
      reloadUrl: ""
      prometheus:
        enabled: false # thanos sidecar takes care of the config reload
        name: configmap-reload # we can disable the prometheus configmap reload
        image:
          repository: redacted-aws-ecr.com/prometheus-config-reloader
          digest: ""
          tag: v0.72.0
    server:
      name: server
      clusterRoleNameOverride: ""
      image:
        repository: redacted-aws-ecr.com/prometheus
        tag: v2.51.2
        digest: ""
        pullPolicy: IfNotPresent
      replicaCount: 2
      retention: 2y
      service:
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "9090"
      statefulSet:
        enabled: true
      podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "10902"
      sidecarContainers:
        thanos-sidecar:
          image: quay.io/thanos/thanos:v0.34.0
          resources:
            requests:
              memory: "512Mi"
          args:
            - "sidecar"
            - "--log.level=debug"
            - "--tsdb.path=/data/"
            - "--prometheus.url=https://prometheus.redacted"
            - "--objstore.config={type: FILESYSTEM, config: {directory: /var/thanos/data}}" # --objstore.config={type: GCS, config: {bucket: BUCKET_REPLACE_ME}}
            - "--reloader.config-file=/etc/prometheus-config/prometheus.yml"
            - "--reloader.config-envsubst-file=/etc/config/prometheus.yml"
            - "--reloader.rule-dir=/etc/prometheus-config/rules"
          ports:
            - name: sidecar-http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: cluster
              containerPort: 10900
          volumeMounts:
            - name: storage-volume
              mountPath: /data
            - name: config-volume
              mountPath: /etc/prometheus-config
              readOnly: false
            - name: prometheus-config-shared
              mountPath: /etc/config/
              readOnly: false
      configPath: /etc/config/prometheus.yml
      persistentVolume:
        enabled: true
        accessModes:
          - ReadWriteOnce
        size: 300Gi
      extraVolumes: # spec.template.spec.volumes
        - name: prometheus-config-shared
          emptyDir: {}
      extraVolumeMounts: # spec.template.spec.containers.volumeMounts for prometheus container
        - name: prometheus-config-shared
          mountPath: /etc/config/
      resources:
        requests:
          memory: 1Gi
      priorityClassName: ""
      enableServiceLinks: true
      extraSecretMounts:
        - name: redacted-certs
          mountPath: /etc/secrets/redactedcerts
          subPath: ""
          secretName: redacted-certs
          readOnly: true
      extraArgs:
        log.level: debug
        storage.tsdb.min-block-duration: 2h # Don't change this, see docs/components/sidecar.md
        storage.tsdb.max-block-duration: 2h
      extraFlags:
        - web.enable-lifecycle
        - web.route-prefix=/
        - web.external-url=https://prometheus.redacted.com/
    alertmanager:
      image:
        repository: redacted-aws-ecr.com/prometheus-alertmanager
        tag: v0.27.0
    kube-state-metrics:
      image:
        registry: redacted-aws-ecr.com
        repository: prometheus-kube-state-metrics
        tag: v2.12.0
    prometheus-node-exporter:
      image:
        registry: redacted-aws-ecr.com
        repository: prometheus-node-exporter
        tag: v1.7.0
    prometheus-pushgateway:
      image:
        repository: redacted-aws-ecr.com/prometheus-pushgateway
        tag: v1.8.0

[dosubot[bot]]

Hey @pritster5! I'm here to help you with your issue. Let's tackle this together!

I found a similar issue that might be relevant to your situation. The error with Thanos Sidecar not being able to upload due to missing external labels can be resolved by ensuring that your Prometheus configuration includes globally unique external labels. Here's an example configuration:

global:
  external_labels:
    cluster: monitoring-test
    replica: 0

Make sure that the external_labels section is correctly configured and includes all necessary labels to uniquely identify your Prometheus instance ^[1].

Regarding the 503 Service Unavailable error, it seems to be related to the --web.enable-lifecycle flag. Ensure that this flag is correctly set in your Prometheus configuration, as it enables the reload endpoint, which is necessary for Thanos Sidecar to function properly ^[2].

Additionally, the "No Healthy Upstream" error might indicate connectivity issues between Thanos Sidecar and Prometheus. Double-check the network configuration and ensure that the Prometheus URL specified in the Sidecar's arguments is correct and accessible.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other | Bug Report}

[pritster5]

The web.enable-lifecycle field is already set. How am I supposed to use global labels if I have multiple replicas configured?

[dosubot[bot]]

When you have multiple Prometheus replicas, it's important to configure global external labels in a way that uniquely identifies each instance. This is crucial for Thanos to properly aggregate and deduplicate data across all instances. You can achieve this by adding a replica label to your external labels configuration. Here's an example:

global:
  external_labels:
    region: eu-west
    monitor: infrastructure
    replica: A

In this setup, the replica label should be unique for each Prometheus instance (e.g., replica: A, replica: B, etc.). This ensures that Thanos can distinguish between different replicas and handle data correctly ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other | Bug Report}

[pritster5]

Adding a global label of the form

global:
        scrape_interval: 5s
        scrape_timeout: 4s
        external_labels:
          prometheus_group: redacted_cluster_name
          prometheus_replica: '$(HOSTNAME)'
        evaluation_interval: 5s

under spec > values > server
allowed the sidecar to start, but we're still seeing errors regarding the lifecycle even though the flag is set, and we can see that the flag is set when describing the pod