Additional kubelet configuration settings

[stmcginnis]

Issue number:

Closes: #2877
Closes: #1792
Closes: #1445
Closes: #2977

Description of changes:

This adds new settings that can be used for soft eviction, graceful shutdown, CPU quota enforcement, and memory manager policy.

The new settings are:

Soft eviction

• settings.kubernetes.eviction-soft
• settings.kubernetes.eviction-soft-grace-period
• settings.kubernetes.eviction-max-pod-grace-period

eviction-soft is the same as the existing eviction-hard but allows a grace period before pods are evicted. The eviction-soft-grace-period allows setting this grace period for each of the support eviction keys. eviction-max-grace-period is used to set an overall maximum for the eviction grace period.

Graceful shutdown

• settings.kubernetes.shutdown-grace-period
• settings.kubernetes.shutdown-grace-period-for-critical-pods

These allow setting a grace period to wait on node shut down to allow pods to exit. Of that grace period, a portion can be dedicated to making sure critical pods are shut down.

CPU quota enforcement

• settings.kubernetes.cpu-cfs-quota-enforced

This sets a flag telling kubelet whether or not to enforce quota for containers that specify CPU limits. The default behavior is to enforce these quotas.

For additional reference on the kubelet setting values, see the node eviction documentation and the KubeletConfiguration docs.

Memory manager

• settings.kubernetes.memory-manager-policy
• settings.kubernetes.memory-manager-reserved-memory

Allows configuring kubelet settings for the memory manager.

Testing done:

Built and deployed a cluster. Configured these settings with:

apiclient apply << EOF
[settings.kubernetes.eviction-hard]
"memory.available" = "15%"

[settings.kubernetes.eviction-soft]
"memory.available" = "12%"

[settings.kubernetes.eviction-soft-grace-period]
"memory.available" = "30s"

[settings.kubernetes]
cpu-cfs-quota-enforced = false
shutdown-grace-period = "30s"
shutdown-grace-period-for-critical-pods = "10s"
eviction-max-pod-grace-period = 40
memory-manager-policy = "Static"

[settings.kubernetes.memory-manager-reserved-memory.0]
enabled = true
memory = "674Mi"
EOF

Verified values were accepted. The performed:

cat /etc/kubernetes/kubelet/config

Verified the config file had the expected configuration keys and values. Then ran:

systemctl status kubelet
journalctl -u kubelet

And made sure there were no errors or issues with service operation.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[jpculp]

Should #2742 be part of this as well?

[stmcginnis]

Should #2742 be part of this as well?

My initial reaction was yes, but after reading through that some more, I think it might be safer to keep that one separate. It has some special casing since it's only applicable to 1.25 and later.

I think if this one merges first, that one could just get added to the same migration if it ends up being in the same release cycle. But I think it would be better to keep that separate in its own PR.

[kahirokunn]

I am very happy with this PR as I just needed a shutdown-grace-period !

[stmcginnis]

Rebased to resolve merge conflicts.

[stmcginnis]

Added memory manager policy and settings.

[stmcginnis]

Rebased to address conflict in Release.toml due to 1.13.3 and 1.13.4.

[zmrow]

LGTM!

[sanjeevpandey19]

Hi @stmcginnis, First of all, thanks for this PR. Just out of curiosity, wanted to know if this is going to be part of the 1.14.0 release.

[stmcginnis]

Hi @stmcginnis, First of all, thanks for this PR. Just out of curiosity, wanted to know if this is going to be part of the 1.14.0 release.

Hi @sanjeevpandey19! Yes, hoping to have this part of the 1.14.0 release. Everything is subject to change based on what is found and testing and other issues of course, but hoping to have this in a 1.14.0 release in the very near future. ;)