deletionThreadPool not closed in LeaseCleanupManager

[yutian-wu]

Greetings,

As we were using the KCL 2.3 recently, we noticed that even after the scheduler has been shut down, part of the LeaseCleanupManager is still running, continuously logging this message:

Number of pending leases to clean before the scan : 0

After looking into the source code along with thread dump, I found that deletionThreadPool in LeaseCleanupManager is never closed.

deletionThreadPool.scheduleAtFixedRate(
  new LeaseCleanupThread(), INITIAL_DELAY, leaseCleanupIntervalMillis, TimeUnit.MILLISECONDS
);

More context at:

/~https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/leases/LeaseCleanupManager.java#L88

So I am creating this issue to have the problem tracked.

Thanks,

[turcsanyip]

I ran into the same issue with KCL 1.
1.14.x cannot stop cleanly but LeaseCleanupManager keeps running forever.
Versions up to 1.13.3 can stop properly.

As far as I can see, the root cause is this:

amazon-kinesis-client/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java

Lines 579 to 580 in f38dd18

	this.leaseCleanupManager = LeaseCleanupManager.newInstance(streamConfig.getStreamProxy(), leaseCoordinator.getLeaseManager(),
	Executors.newSingleThreadScheduledExecutor(), metricsFactory, cleanupLeasesUponShardCompletion,

The Worker creates a ScheduledExecutorService for LeaseCleanupManager on the fly. As there is no reference to the executor, it cannot be stopped later.