Panic if Elasticsearch is unavailable during shutdown of the exporter

[lahsivjar]

Component(s)

exporter/elasticsearch

What happened?

Description

If Elasticsearch is unavailable on shutdown of the exporter then if there is data to be flushed in the exporter it will cause a panic.

Steps to Reproduce

1. Shutdown Elasticsearch
2. Keep sending data to the collector configured with Elasticsearch exporter
3. Shutdown the exporter

Expected Result

No panic is observed

Actual Result

Panic is observed:

panic: send on closed channel

goroutine 78 [running]:
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).Add(0x14000334a80, {0x102586420, 0x1033c5460}, {{0x101f86420, 0x14}, {0x101f70693, 0x6}, {0x0, 0x0}, {0x0, ...}, ...})
...

The root cause of the panic is the handling of bulk failures (ref). When the item fails then we try to push the item back into the bulk indexer, however, if the exporter is shutting down then the bulkindexer would already be closed and the flush (which led to the error) is due to ES not available on the final flush initiated by Close.

Collector version

all

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
          otlp:
            protocols:
              grpc:
                endpoint: "127.0.0.1:60979"
        exporters:
          elasticsearch:
            endpoints: [http://127.0.0.1:9200]
            flush:
              interval: 1s
            sending_queue:
              enabled: true
            retry:
              enabled: true
              max_requests: 10000

          debug:
            verbosity: basic

        processors:


        extensions:


        service:
          telemetry:
            metrics:
              address: 127.0.0.1:60980
          extensions: []
          pipelines:
            logs:
              receivers: [otlp]
              processors: []
              exporters: [elasticsearch]

Log output

panic: send on closed channel

goroutine 78 [running]:
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).Add(0x14000334a80, {0x102586420, 0x1033c5460}, {{0x101f86420, 0x14}, {0x101f70693, 0x6}, {0x0, 0x0}, {0x0, ...}, ...})

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ycombinator]

/label help-wanted

[ycombinator]

/label -needs-triage

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[carsonip]

This should not be the case now, as we have migrated to go-docappender and that the OnFailure handler is no longer used.