Kubernetes graceful shutdown not working as expected

[stmcginnis]

tldr: Kubernetes graceful shutdown relies on inhibitor locks. These are part of systemd-logind, which is not currently included as part of Bottlerocket.

Discussed in #3291

^{Originally posted by carlosjgp July 25, 2023}
I've been working on a way to roll out our EKS nodes gracefully and my first attempt was trying to use K8s native support for node graceful shutdown and avoid using any extra infrastructure or deployments

I can see that support for its properties here

• Support for Graceful Node Shutdown in the Kubelet Configuration #1792
• Additional kubelet configuration settings #2930

I tried setting these to shutdown_grace_period=3m and shutdown_grace_period_critical_pod=2m and swapping the nodes between Bottlerocket version 1.14.1 and 1.14.2 while running a simple Nginx deployment (helm create test) with 8 replicas and ingress setup to accept traffic

Then run vegeta to hit this Nginx deployment to ensure the pods were gracefully moved across and respecting the PodDisruptionBudget but there are a lot of failed requests

The nodes are supposed to be tainted with node.kubernetes.io/not-ready:NoSchedule but I didn't see this happening either

I'm using:

• EKS K8s 1.27
• BottleRocket AMI from SSM parameter /aws/service/bottlerocket/aws-k8s-1.27/x86_64/${var.bottlerocket_version}/image_id
• AWS Autoscaling group Instace refresh to roll out the nodes when the AMI id changes

Has someone seen this before?

I could provide more details from Cloudwatch, Container or OS logs and K8s events