[v10.1.x] Alerting: Do not exit if Redis ping fails when using redis-based Alertmanager clustering

[grafana-delivery-bot]

Backport 5c9aeae from #74144

---

What is this feature?

The normal gossip-based alertmanager clustering is tolerant to network problems. The peers might not be able to reach each other, but Grafana will still run, albeit without deduplication for high-availability alerts.

grafana/pkg/services/ngalert/notifier/multiorg_alertmanager.go

Lines 130 to 133 in 64652a9

	err = peer.Join(cluster.DefaultReconnectInterval, cluster.DefaultReconnectTimeout)
	if err != nil {
	moa.logger.Error("msg", "Unable to join gossip mesh while initializing cluster for high availability mode", "error", err)
	}

(note the error is ignored)

Most of the Redis-based clustering also works this way most of the time - one exception is the ping that happens when the connection is first created. Failure to ping causes the Grafana process to exit.

Given we heavily favor availability for notifications, this PR fixes the Redis implementation to follow the same failure semantics as gossip.

The trade-off is that genuinely bad redis configurations are now harder to discover. It's consistent with other HA configurations, though.

Why do we need this feature?

Prevents a failure mode of the system in favor of allowing duplicate notifications.

Which issue(s) does this PR fix?:

n/a

Special notes for your reviewer:

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.