Keepalived not issuing keepalive failures in all cases

[echlebek]

While doing QA on beta 6, keepalive failures were not always observed as expected. The behaviour appears to be stochastic.

Expected Behavior

When an agent with deregister: false is stopped, keepalive failures should occur after the keepalive timeout.

Current Behavior

Sometimes keepalive failures do occur, and sometimes they do not.

Steps to Reproduce (for bugs)

1. Stop an agent
2. Observe keepalive failure does not occur

Context

QA on sensu beta 6

Your Environment

Ubuntu

[palourde]

Might be related to #2077.

Some relevant information from the staging environment.

On backend-2:

Oct 19 15:10:20 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"debug","msg":"starting a monitor for the key /sensu.io/monitors/staging-agent-0","time":"2018-10-19T15:10:20Z"}
Oct 19 15:10:20 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"info","msg":"shutting down monitor for /sensu.io/monitors/staging-agent-0","time":"2018-10-19T15:10:20Z"}
Oct 19 15:10:28 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"debug","msg":"starting a monitor for the key /sensu.io/monitors/staging-agent-0","time":"2018-10-19T15:10:28Z"}
Oct 19 15:10:28 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"info","msg":"shutting down monitor for /sensu.io/monitors/staging-agent-0","time":"2018-10-19T15:10:28Z"}
Oct 19 15:10:48 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"debug","msg":"a lease for the key /sensu.io/monitors/staging-agent-0 already exist, extending it","time":"2018-10-19T15:10:48Z"}
Oct 19 15:11:08 ip-10-0-1-116 sensu-backend[3858]: {"component":"monitor","level":"debug","msg":"a lease for the key /sensu.io/monitors/staging-agent-0 already exist, extending it","time":"2018-10-19T15:11:08Z"}

The lease is only revoked in

sensu-go/backend/monitor/supervisor.go

Line 174 in f0d39c1

if _, err := cli.Lease.Revoke(ctx, mon.leaseID); err != nil {

The fact that the monitor starts and stops immediately might provoke some weird corner case.

[barryorourke]

This could be related to #2135