Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional kubelet configuration settings #2930

Merged
merged 6 commits into from
May 4, 2023
Merged

Additional kubelet configuration settings #2930

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bf09aeb
kubelet: Add `cpu-cfs-quota-enforced` setting
stmcginnis Mar 22, 2023
00381cf
kubelet: Add shutdown grace period settings
stmcginnis Mar 22, 2023
fc02459
kubelet: Add soft eviction settings
stmcginnis Mar 22, 2023
9e76476
settings: Enabling constraining i32 to non-negative value
stmcginnis Mar 22, 2023
ac1e6e9
kubelet: Add reserved memory settings
stmcginnis Apr 5, 2023
9f93779
kubelet: Add aliases for ValidCpuManagerPolicy
stmcginnis May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 60 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,7 @@ The following settings are optional and allow you to further configure your clus
* `settings.kubernetes.cluster-domain`: The DNS domain for this cluster, allowing all Kubernetes-run containers to search this domain before the host's search domains. Defaults to `cluster.local`.
* `settings.kubernetes.container-log-max-files`: The maximum number of container log files that can be present for a container.
* `settings.kubernetes.container-log-max-size`: The maximum size of container log file before it is rotated.
* `settings.kubernetes.cpu-cfs-quota-enforced`: Whether CPU CFS quotas are enforced. Defaults to `true`.
* `settings.kubernetes.cpu-manager-policy`: Specifies the CPU manager policy. Possible values are `static` and `none`. Defaults to `none`. If you want to allow pods with certain resource characteristics to be granted increased CPU affinity and exclusivity on the node, you can set this setting to `static`. You should reboot if you change this setting after startup - try `apiclient reboot`.
* `settings.kubernetes.cpu-manager-policy-options`: Policy options to apply when `cpu-manager-policy` is set to `static`. Currently `full-pcpus-only` is the only option.

Expand Down Expand Up @@ -476,47 +477,90 @@ The following settings are optional and allow you to further configure your clus
* `settings.kubernetes.event-burst`: The maximum size of a burst of event creations.
* `settings.kubernetes.event-qps`: The maximum event creations per second.
* `settings.kubernetes.eviction-hard`: The signals and thresholds that trigger pod eviction.
* `settings.kubernetes.eviction-max-pod-grace-period`: Maximum grace period, in seconds, to wait for pod termination before soft eviction. Default is `0`.
* `settings.kubernetes.eviction-soft`: The signals and thresholds that trigger pod eviction with a provided grace period.
* `settings.kubernetes.eviction-soft-grace-period`: Delay for each signal to wait for pod termination before eviction.

Remember to quote signals (since they all contain ".") and to quote all values.

Example user data for setting up eviction hard:
Example user data for setting up eviction values:

```toml
[settings.kubernetes.eviction-hard]
"memory.available" = "15%"

[settings.kubernetes.eviction-soft]
"memory.available" = "12%"

[settings.kubernetes.eviction-soft-grace-period]
"memory.available" = "30s"

[settings.kubernetes]
"eviction-max-pod-grace-period" = 40
```

* `settings.kubernetes.image-gc-high-threshold-percent`: The percent of disk usage after which image garbage collection is always run, expressed as an integer from 0-100 inclusive.
* `settings.kubernetes.image-gc-low-threshold-percent`: The percent of disk usage before which image garbage collection is never run, expressed as an integer from 0-100 inclusive.

Since v1.14.0 `image-gc-high-threshold-percent` and `image-gc-low-threshold-percent` can be represented as numbers.
For example:
Since v1.14.0 `image-gc-high-threshold-percent` and `image-gc-low-threshold-percent` can be represented as numbers.
For example:

```toml
[settings.kubernetes]
image-gc-high-threshold-percent = 85
image-gc-low-threshold-percent = 80
```
```toml
[settings.kubernetes]
image-gc-high-threshold-percent = 85
image-gc-low-threshold-percent = 80
```

For backward compatibility, both string and numeric representations are accepted since v1.14.0.
Prior to v1.14.0 these needed to be represented as strings, for example:
For backward compatibility, both string and numeric representations are accepted since v1.14.0.
Prior to v1.14.0 these needed to be represented as strings, for example:

```toml
[settings.kubernetes]
image-gc-high-threshold-percent = "85"
image-gc-low-threshold-percent = "80"
```
```toml
[settings.kubernetes]
image-gc-high-threshold-percent = "85"
image-gc-low-threshold-percent = "80"
```

If you downgrade from v1.14.0 to an earlier version, and you have these values set as numbers, they will be converted to strings on downgrade.
If you downgrade from v1.14.0 to an earlier version, and you have these values set as numbers, they will be converted to strings on downgrade.

* `settings.kubernetes.kube-api-burst`: The burst to allow while talking with kubernetes.
* `settings.kubernetes.kube-api-qps`: The QPS to use while talking with kubernetes apiserver.
* `settings.kubernetes.log-level`: Adjust the logging verbosity of the `kubelet` process.
The default log level is 2, with higher numbers enabling more verbose logging.
* `settings.kubernetes.memory-manager-policy`: The memory management policy to use: `None` (default) or `Static`.
stmcginnis marked this conversation as resolved.
Show resolved Hide resolved
Note, when using the `Static` policy you should also set `settings.kubernetes.memory-manager-reserved-memory` values.
* `settings.kubernetes.memory-manager-reserved-memory`: Used to set the total amount of reserved memory for a node.
These settings are used to configure memory manager policy when `settings.kubernetes.memory-manager-policy` is set to `Static`.

`memory-manager-reserved-memory` is set per NUMA node. For example:

```toml
[settings.kubernetes]
"memory-manager-policy" = "Static"

[settings.kubernetes.memory-manager-reserved-memory.0]
# Reserve a single 1GiB huge page along with 674MiB of memory
"enabled" = true
"memory" = "674Mi"
"hugepages-1Gi" = "1Gi"

[settings.kubernetes.memory-manager-reserved-memory.1]
# Reserve 1,074 2MiB huge pages
"enabled" = true
"hugepages-2Mi" = "2148Mi"
```

**Warning:** `memory-manager-reserved-memory` settings are an advanced configuration and requires a clear understanding of what you are setting.
Misconfiguration of reserved memory settings may cause the Kubernetes `kubelet` process to fail.
It can be very difficult to recover from configuration errors.
Use the memory reservation information from `kubectl describe node` and make sure you understand the Kubernetes documentation related to the [memory manager](https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/) and how to [reserve compute resources for system daemons](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/).

* `settings.kubernetes.pod-pids-limit`: The maximum number of processes per pod.
* `settings.kubernetes.provider-id`: This sets the unique ID of the instance that an external provider (i.e. cloudprovider) can use to identify a specific node.
* `settings.kubernetes.registry-burst`: The maximum size of bursty pulls.
* `settings.kubernetes.registry-qps`: The registry pull QPS.
* `settings.kubernetes.server-tls-bootstrap`: Enables or disables server certificate bootstrap. When enabled, the kubelet will request a certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests (CSR). Defaults to `true`.
* `settings.kubernetes.shutdown-grace-period`: Delay the node should wait for pod termination before shutdown. Default is `0s`.
* `settings.kubernetes.shutdown-grace-period-for-critical-pods`: The portion of the shutdown delay that should be dedicated to critical pod shutdown. Default is `0s`.
* `settings.kubernetes.standalone-mode`: Whether to run the kubelet in standalone mode, without connecting to an API server. Defaults to `false`.
* `settings.kubernetes.system-reserved`: Resources reserved for system components.

Expand Down
1 change: 1 addition & 0 deletions Release.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,5 +204,6 @@ version = "1.14.0"
"(1.13.5, 1.14.0)" = [
"migrate_v1.14.0_kubernetes-gc-percent-type-change.lz4",
"migrate_v1.14.0_kubelet-config-settings.lz4",
"migrate_v1.14.0_kubelet-prefix-config-settings.lz4",
"migrate_v1.14.0_k8s-services-mode.lz4",
]
45 changes: 45 additions & 0 deletions packages/kubernetes-1.22/kubelet-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,21 @@ evictionHard:
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft}}
evictionSoft:
{{#each settings.kubernetes.eviction-soft}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft-grace-period}}
evictionSoftGracePeriod:
{{#each settings.kubernetes.eviction-soft-grace-period}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-max-pod-grace-period}}
evictionMaxPodGracePeriod: {{settings.kubernetes.eviction-max-pod-grace-period}}
{{/if}}
{{#if settings.kubernetes.allowed-unsafe-sysctls}}
allowedUnsafeSysctls: {{settings.kubernetes.allowed-unsafe-sysctls}}
{{/if}}
Expand Down Expand Up @@ -76,6 +91,7 @@ systemReserved:
{{/each}}
systemReservedCgroup: "/system"
{{/if}}
cpuCFSQuota: {{default true settings.kubernetes.cpu-cfs-quota-enforced}}
cpuManagerPolicy: {{default "none" settings.kubernetes.cpu-manager-policy}}
{{#if settings.kubernetes.cpu-manager-reconcile-period}}
cpuManagerReconcilePeriod: {{settings.kubernetes.cpu-manager-reconcile-period}}
Expand Down Expand Up @@ -133,3 +149,32 @@ containerLogMaxSize: {{settings.kubernetes.container-log-max-size}}
{{#if settings.kubernetes.container-log-max-files includeZero=true}}
containerLogMaxFiles: {{settings.kubernetes.container-log-max-files}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period}}
shutdownGracePeriod: {{settings.kubernetes.shutdown-grace-period}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period-for-critical-pods}}
shutdownGracePeriodCriticalPods: {{settings.kubernetes.shutdown-grace-period-for-critical-pods}}
{{/if}}
{{#if settings.kubernetes.memory-manager-reserved-memory}}
{{#if (any_enabled settings.kubernetes.memory-manager-reserved-memory)}}
{{#if settings.kubernetes.memory-manager-policy}}
memoryManagerPolicy: {{settings.kubernetes.memory-manager-policy}}
{{/if}}
reservedMemory:
{{#each settings.kubernetes.memory-manager-reserved-memory}}
{{#if this.enabled}}
- numaNode: {{@key}}
limits:
{{#if this.memory}}
memory: {{this.memory}}
{{/if}}
{{#if this.hugepages-1Gi}}
hugepages-1Gi: {{this.hugepages-1Gi}}
{{/if}}
{{#if this.hugepages-2Mi}}
hugepages-2Mi: {{this.hugepages-2Mi}}
{{/if}}
{{/if}}
{{/each}}
{{/if}}
{{/if}}
45 changes: 45 additions & 0 deletions packages/kubernetes-1.23/kubelet-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,21 @@ evictionHard:
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft}}
evictionSoft:
{{#each settings.kubernetes.eviction-soft}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft-grace-period}}
evictionSoftGracePeriod:
{{#each settings.kubernetes.eviction-soft-grace-period}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-max-pod-grace-period}}
evictionMaxPodGracePeriod: {{settings.kubernetes.eviction-max-pod-grace-period}}
{{/if}}
{{#if settings.kubernetes.allowed-unsafe-sysctls}}
allowedUnsafeSysctls: {{settings.kubernetes.allowed-unsafe-sysctls}}
{{/if}}
Expand Down Expand Up @@ -76,6 +91,7 @@ systemReserved:
{{/each}}
systemReservedCgroup: "/system"
{{/if}}
cpuCFSQuota: {{default true settings.kubernetes.cpu-cfs-quota-enforced}}
cpuManagerPolicy: {{default "none" settings.kubernetes.cpu-manager-policy}}
{{#if settings.kubernetes.cpu-manager-reconcile-period}}
cpuManagerReconcilePeriod: {{settings.kubernetes.cpu-manager-reconcile-period}}
Expand Down Expand Up @@ -135,3 +151,32 @@ containerLogMaxSize: {{settings.kubernetes.container-log-max-size}}
{{#if settings.kubernetes.container-log-max-files includeZero=true}}
containerLogMaxFiles: {{settings.kubernetes.container-log-max-files}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period}}
shutdownGracePeriod: {{settings.kubernetes.shutdown-grace-period}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period-for-critical-pods}}
shutdownGracePeriodCriticalPods: {{settings.kubernetes.shutdown-grace-period-for-critical-pods}}
{{/if}}
{{#if settings.kubernetes.memory-manager-reserved-memory}}
{{#if (any_enabled settings.kubernetes.memory-manager-reserved-memory)}}
{{#if settings.kubernetes.memory-manager-policy}}
memoryManagerPolicy: {{settings.kubernetes.memory-manager-policy}}
{{/if}}
reservedMemory:
{{#each settings.kubernetes.memory-manager-reserved-memory}}
{{#if this.enabled}}
- numaNode: {{@key}}
limits:
{{#if this.memory}}
memory: {{this.memory}}
{{/if}}
{{#if this.hugepages-1Gi}}
hugepages-1Gi: {{this.hugepages-1Gi}}
{{/if}}
{{#if this.hugepages-2Mi}}
hugepages-2Mi: {{this.hugepages-2Mi}}
{{/if}}
{{/if}}
{{/each}}
{{/if}}
{{/if}}
45 changes: 45 additions & 0 deletions packages/kubernetes-1.24/kubelet-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,21 @@ evictionHard:
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft}}
evictionSoft:
{{#each settings.kubernetes.eviction-soft}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft-grace-period}}
evictionSoftGracePeriod:
{{#each settings.kubernetes.eviction-soft-grace-period}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-max-pod-grace-period}}
evictionMaxPodGracePeriod: {{settings.kubernetes.eviction-max-pod-grace-period}}
{{/if}}
{{#if settings.kubernetes.allowed-unsafe-sysctls}}
allowedUnsafeSysctls: {{settings.kubernetes.allowed-unsafe-sysctls}}
{{/if}}
Expand Down Expand Up @@ -76,6 +91,7 @@ systemReserved:
{{/each}}
systemReservedCgroup: "/system"
{{/if}}
cpuCFSQuota: {{default true settings.kubernetes.cpu-cfs-quota-enforced}}
cpuManagerPolicy: {{default "none" settings.kubernetes.cpu-manager-policy}}
{{#if settings.kubernetes.cpu-manager-reconcile-period}}
cpuManagerReconcilePeriod: {{settings.kubernetes.cpu-manager-reconcile-period}}
Expand Down Expand Up @@ -134,3 +150,32 @@ containerLogMaxSize: {{settings.kubernetes.container-log-max-size}}
{{#if settings.kubernetes.container-log-max-files includeZero=true}}
containerLogMaxFiles: {{settings.kubernetes.container-log-max-files}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period}}
shutdownGracePeriod: {{settings.kubernetes.shutdown-grace-period}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period-for-critical-pods}}
shutdownGracePeriodCriticalPods: {{settings.kubernetes.shutdown-grace-period-for-critical-pods}}
{{/if}}
{{#if settings.kubernetes.memory-manager-reserved-memory}}
{{#if (any_enabled settings.kubernetes.memory-manager-reserved-memory)}}
{{#if settings.kubernetes.memory-manager-policy}}
memoryManagerPolicy: {{settings.kubernetes.memory-manager-policy}}
{{/if}}
reservedMemory:
{{#each settings.kubernetes.memory-manager-reserved-memory}}
{{#if this.enabled}}
- numaNode: {{@key}}
limits:
{{#if this.memory}}
memory: {{this.memory}}
{{/if}}
{{#if this.hugepages-1Gi}}
hugepages-1Gi: {{this.hugepages-1Gi}}
{{/if}}
{{#if this.hugepages-2Mi}}
hugepages-2Mi: {{this.hugepages-2Mi}}
{{/if}}
{{/if}}
{{/each}}
{{/if}}
{{/if}}
45 changes: 45 additions & 0 deletions packages/kubernetes-1.25/kubelet-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,21 @@ evictionHard:
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft}}
evictionSoft:
{{#each settings.kubernetes.eviction-soft}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-soft-grace-period}}
evictionSoftGracePeriod:
{{#each settings.kubernetes.eviction-soft-grace-period}}
{{@key}}: "{{this}}"
{{/each}}
{{/if}}
{{#if settings.kubernetes.eviction-max-pod-grace-period}}
evictionMaxPodGracePeriod: {{settings.kubernetes.eviction-max-pod-grace-period}}
{{/if}}
{{#if settings.kubernetes.allowed-unsafe-sysctls}}
allowedUnsafeSysctls: {{settings.kubernetes.allowed-unsafe-sysctls}}
{{/if}}
Expand Down Expand Up @@ -76,6 +91,7 @@ systemReserved:
{{/each}}
systemReservedCgroup: "/system"
{{/if}}
cpuCFSQuota: {{default true settings.kubernetes.cpu-cfs-quota-enforced}}
cpuManagerPolicy: {{default "none" settings.kubernetes.cpu-manager-policy}}
{{#if settings.kubernetes.cpu-manager-reconcile-period}}
cpuManagerReconcilePeriod: {{settings.kubernetes.cpu-manager-reconcile-period}}
Expand Down Expand Up @@ -134,3 +150,32 @@ containerLogMaxSize: {{settings.kubernetes.container-log-max-size}}
{{#if settings.kubernetes.container-log-max-files includeZero=true}}
containerLogMaxFiles: {{settings.kubernetes.container-log-max-files}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period}}
shutdownGracePeriod: {{settings.kubernetes.shutdown-grace-period}}
{{/if}}
{{#if settings.kubernetes.shutdown-grace-period-for-critical-pods}}
shutdownGracePeriodCriticalPods: {{settings.kubernetes.shutdown-grace-period-for-critical-pods}}
{{/if}}
{{#if settings.kubernetes.memory-manager-reserved-memory}}
{{#if (any_enabled settings.kubernetes.memory-manager-reserved-memory)}}
{{#if settings.kubernetes.memory-manager-policy}}
memoryManagerPolicy: {{settings.kubernetes.memory-manager-policy}}
{{/if}}
reservedMemory:
{{#each settings.kubernetes.memory-manager-reserved-memory}}
{{#if this.enabled}}
- numaNode: {{@key}}
limits:
{{#if this.memory}}
memory: {{this.memory}}
{{/if}}
{{#if this.hugepages-1Gi}}
hugepages-1Gi: {{this.hugepages-1Gi}}
{{/if}}
{{#if this.hugepages-2Mi}}
hugepages-2Mi: {{this.hugepages-2Mi}}
{{/if}}
{{/if}}
{{/each}}
{{/if}}
{{/if}}
Loading