containerd: add OCI default spec and settings #2697

mchaker · 2023-01-03T22:57:43Z

Issue number:

Description of changes:

This PR adds support for setting default OCI spec values. The default OCI spec stored at /etc/containerd/cri-base.json is applied to all containers run by containerd, unless otherwise configured (e.g. by Kubernetes).

The OCI spec can be found in detail at /~https://github.com/opencontainers/runtime-spec/blob/main/config.md .

This PR also adds some default settings that we want Bottlerocket to ship with: namely, a specific set of process capabilities (sources/models/shared-defaults/oci-capabilities.toml) and resource limits (sources/models/shared-defaults/oci-resource-limits.toml).

Testing done:

OCI Defaults Test Plan

Manual Testing

Baseline

Baseline with Production AMI v1.11 (i.e. ami-09d516ec16c94c0e0)
- Get the OCI spec for a sample nginx workload container launched via kubernetes (save it for comparison)
- Run a pod that opens as many files as it can, as well as printing the container capabilities
  - How many files were opened: 65533 due to our containerd patch.
  - Bounding capabilities: {CAP_SETFCAP, CAP_NET_RAW, CAP_KILL, CAP_CHOWN, CAP_SETUID, CAP_FSETID, CAP_FOWNER, CAP_AUDIT_WRITE, CAP_MKNOD, CAP_SYS_CHROOT, CAP_DAC_OVERRIDE, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_SETGID}

Test case: feature enabled and configured with default settings

Make sure there is no behavior change when not configuring the new settings.

Run modified AMI (ami-0f58fc68a762109e8, 7ad3110d) without setting oci-defaults.capabilities, oci-defaults.resource-limits in userdata
- Get the OCI spec for a nginx workload container: it matched v1.11
- Run a pod that opens as many files as it can, as well as printing the container capabilities
  - How many files were opened: 65533 (matched v1.11)
  - Bounding capabilities: {CAP_CHOWN, CAP_MKNOD, CAP_SYS_CHROOT, CAP_NET_BIND_SERVICE, CAP_SETFCAP, CAP_SETUID, CAP_FOWNER, CAP_DAC_OVERRIDE, CAP_KILL, CAP_SETGID, CAP_FSETID, CAP_SETPCAP, CAP_AUDIT_WRITE} (matched v1.11, with the exception of CAP_NET_RAW, which is intentionally removed)
Log in to the test node and verify that the cri-base.json file contains the desired settings default values

Test case: set capabilities

Run modified AMI (ami-01b0018dc3c2ad0de, 3940728)
- Add a capability (CAP_BPF) via apiclient
Run a pod that opens as many files as it can, as well as printing the container capabilities
- How many files were opened: 65533 (unchaged from defaults)
- Bounding capabilities: {CAP_SETFCAP, CAP_MKNOD, CAP_SETUID, CAP_BPF, CAP_SETGID, CAP_FSETID, CAP_AUDIT_WRITE, CAP_NET_BIND_SERVICE, CAP_DAC_OVERRIDE, CAP_CHOWN, CAP_KILL, CAP_FOWNER, CAP_SETPCAP, CAP_SYS_CHROOT} (capabilities changed accordingly: CAP_BPF was added)
Log in to the test node and verify that the cri-base.json file contains the desired settings values

Test case: set resource-limits

Run modified AMI (ami-01b0018dc3c2ad0de, 3940728)
- Use apiclient to change the RLIMIT_NOFILE soft value to 90000
Run a pod that opens as many files as it can, as well as printing the container capabilities
- How many files were opened: 89997 (changed according to new RLIMIT_NOFILE soft)
- Bounding capabilities: {CAP_SETPCAP, CAP_FOWNER, CAP_AUDIT_WRITE, CAP_CHOWN, CAP_SETGID, CAP_MKNOD, CAP_SYS_CHROOT, CAP_SETFCAP, CAP_KILL, CAP_NET_BIND_SERVICE, CAP_FSETID, CAP_SETUID, CAP_DAC_OVERRIDE} (unchanged from defaults)
Log in to the test node and verify that the cri-base.json file contains the desired settings values

Test case: set both capabilities and resource-limits

Combine both of the above: i.e. set capabilities and rlimit_nofile values

Run modified AMI (ami-0628b6b13d2b7c83e, 69c158f)
- Use apiclient to change the RLIMIT_NOFILE soft value to 90000
- Use apiclient to add a capability (CAP_BPF)
Run a pod that opens as many files as it can, as well as printing the container capabilities
- How many files were opened: 89997 (changed according to new RLIMIT_NOFILE soft)
- Bounding capabilities: {CAP_KILL, CAP_SETUID, CAP_NET_BIND_SERVICE, CAP_DAC_OVERRIDE, CAP_BPF, CAP_SETPCAP, CAP_AUDIT_WRITE, CAP_MKNOD, CAP_SETFCAP, CAP_SYS_CHROOT, CAP_CHOWN, CAP_SETGID, CAP_FSETID, CAP_FOWNER} (capabilities changed accordingly: CAP_BPF was added)
Log in to the test node and verify that the cri-base.json file contains the desired settings values

Test all variants

Repeat above Manual Testing steps for the following variants:

aws-k8s-1.23
aws-k8s-1.23-nvidia
metal-k8s-1.24
vmware-k8s-1.23

Run abridged Manual Testing steps for the following variants:

Run Sonobuoy Conformance

Run Sonobuoy quick suite
Run Sonobuoy full suite

Test case: Upgrade Downgrade

v1.11 --> this version --> v1.11

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

mchaker

remaining TODOs based on early feedback

sources/models/shared-defaults/oci-capabilities.toml

sources/models/shared-defaults/oci-defaults.toml

sources/models/shared-defaults/oci-resource-limits.toml

sources/models/src/modeled_types/oci_defaults.rs

sources/models/shared-defaults/oci-capabilities.toml

mchaker · 2023-01-03T23:57:30Z

Force push: resolved remaining TODOs based on early feedback.

mchaker · 2023-01-04T00:56:14Z

Force push: resolved merge conflict

sources/api/datastore/src/serialization/mod.rs

mchaker · 2023-01-05T14:56:21Z

Force push: removed panic in helper and reset default capabilities to 1.0.2-dev defaults.

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/Cargo.toml

README.md

sources/models/src/modeled_types/mod.rs

sources/api/schnauzer/src/helpers.rs

mchaker · 2023-01-05T23:50:06Z

Force push: address some comments

webern · 2023-01-06T00:21:06Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/e19d8eba27b22fedc5d253657899cd8d72d5d4ca..c633a175eadb7edd95f6fb1374de3ef16d8b62d8

break out the migration to its own commit

webern · 2023-01-06T01:10:25Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/c633a175eadb7edd95f6fb1374de3ef16d8b62d8..46d0035f43b35102443877f2338a763f2d8fedec

Use #[model(add_option = false)] to require both hard and soft limits to be set (as is required by the spec).
Add the OCI defaults settings to all the k8s variants.

webern · 2023-01-06T01:16:27Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/46d0035f43b35102443877f2338a763f2d8fedec..9d7001a575e61f0ee4cd246845f08d988a1672c4

Fix a small nit.

README.md

mchaker · 2023-01-06T16:48:54Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/9d7001a575e61f0ee4cd246845f08d988a1672c4..bf2b51bcb9b19109de89eceb060496a8b5c1f552

Force push: fix a small typo

bcressey · 2023-01-06T21:52:43Z

README.md

+
+The **default** capabilities that are set in Bottlerocket are:
+
+setting key | setting value


nit: maybe

Suggested change

setting key | setting value

setting | default value

I agree it's clearer - accepted suggestion

bcressey · 2023-01-06T21:56:56Z

README.md

+setting key | corresponding linux process capability
+----- | -----
+`settings.oci-defaults.capabilities.audit-control` | `CAP_AUDIT_CONTROL`
+`settings.oci-defaults.capabilities.audit-read` | `CAP_AUDIT_READ`
+`settings.oci-defaults.capabilities.audit-write` | `CAP_AUDIT_WRITE`
+`settings.oci-defaults.capabilities.block-suspend` | `CAP_BLOCK_SUSPEND`
+`settings.oci-defaults.capabilities.bpf` | `CAP_BPF`
+`settings.oci-defaults.capabilities.checkpoint-restore` | `CAP_CHECKPOINT_RESTORE`
+`settings.oci-defaults.capabilities.chown` | `CAP_CHOWN`


I'd prefer to organize this as:

capability | setting | default value CAP_AUDIT_CONTROL | settings.oci-defaults.capabilities.audit-control | false

With all the default-true ones first, followed by the others. Then we wouldn't have the duplication, and the sorting would be similar to runc's list of capabilities.

What would be preferable for default value in the case of capabilities we do not configure with default values?

A: The semantically correct setting of false
B: The syntactically correct "setting" of (unset)

Also, does this comment (reorganizing to capability | setting | default value) also apply to rlimits as well?
Meaning, reorganizing the rlimits section to be resource limit | setting | default value?

I would prefer something like - to indicate unset, so there's less visual noise.

Making the same change to the rlimits section would be good too.

Updated in the latest force push: /~https://github.com/bottlerocket-os/bottlerocket/compare/52cfccb27a6dfe636699d64696b3adf4f3346625..66c11153cc3521157c6d03a93e049fa1e88af43a

bcressey · 2023-01-06T22:01:26Z

README.md

+##### OCI Defaults: Resource Limits
+
+Each of the `resource-limits` settings below contain two numeric fields: `hard-limit` and `soft-limit`, which are **32-bit unsigned integers**.
+
+Please see the [`getrlimit` linux manpage](https://man7.org/linux/man-pages/man7/capabilities.7.html) for meanings of `hard-limit` and `soft-limit`.


This is quite a lot of change to land in one commit. The functionality is great, no complaints there, but it makes it a bit harder for reviewers to digest.

As a suggestion for commit structure in the future, I'd recommend:

one commit to add the scaffolding and base functionality

one commit to add settings + template fields for rlimits (smaller)

one commit to add settings + template fields for capabilities (larger)

one commit for required migrations

It's fine for this PR though.

Apologies for the monolithic PR structure and thank you for explaining a better way. I'll keep this comment in my notebook as a reminder for my next PR.

bcressey · 2023-01-07T00:00:25Z

sources/models/src/aws-k8s-1.22-nvidia/defaults.d/95-oci-defaults.toml

@@ -0,0 +1 @@
+../../../shared-defaults/oci-defaults.toml


It's a minor thing, but I'd prefer to have these in the 75-79 or 85-89 range, together with ??-oci-hooks.toml, rather than at the end of the 90 range. That way all the OCI related defaults share the same range.

Symlinks renamed in latest force push

bcressey · 2023-01-07T00:34:45Z

sources/models/shared-defaults/oci-defaults-resource-limits.toml

+[settings.oci-defaults.resource-limits.max-open-files]
+hard-limit = 1048576
+soft-limit = 65536


In preparation for future Docker support, we should give these files a name to keep them distinct:

oci-defaults-containerd-cri.toml

oci-defaults-containerd-cri-resource-limits.toml

oci-defaults-containerd-cri-capabilities.toml

The capabilities will be the same but the resource limits and configuration files will be different.

Accepted suggestion, thank you

bcressey · 2023-01-07T00:44:43Z

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/src/main.rs

+    migrate(common_migrations::AddPrefixesMigration(vec![
+        "settings.oci-defaults",
+    ]))


Suggested change

migrate(common_migrations::AddPrefixesMigration(vec![

"settings.oci-defaults",

]))

migrate(common_migrations::AddPrefixesMigration(vec![

"settings.oci-defaults",

"services.oci-defaults",

"configuration-files.oci-defaults",

]))

Thank you, accepted suggestion

bcressey · 2023-01-07T00:46:12Z

sources/models/shared-defaults/oci-defaults.toml

+[metadata.settings.oci-defaults]
+affected-services = ["oci-defaults", "containerd"]


Need a metadata migration for the affected services.

Added in the latest force push: /~https://github.com/bottlerocket-os/bottlerocket/compare/ec2a9f46efb263d54caee5b2df2f4d9be6c3fad2..1318d9557178011eb609a2ab2ba9165199481e83

bcressey · 2023-01-07T00:55:21Z

sources/models/src/modeled_types/oci_defaults.rs

+/// OciDefaultsCapability specifies which process capabilities are
+/// allowed to be modified in Bottlerocket. The Linux process
+/// capabilities are defined in the Linux manpages (see the man7.org link below).
+/// Default values can be found in: sources/models/shared-defaults/oci-capabilities.toml
+/// OCI spec: /~https://github.com/opencontainers/runtime-spec/blob/c4ee7d12c742ffe806cd9350b6af3b4b19faed6f/config.md#linux-process
+/// Linux kernel capabilities: https://man7.org/linux/man-pages/man7/capabilities.7.html
+///
+/// Default capabilities are specified in the following file:
+/// * sources/models/shared-defaults/oci-capabilities.toml


I'd probably clean up all of these links and just say:

Suggested change

/// OciDefaultsCapability specifies which process capabilities are

/// allowed to be modified in Bottlerocket. The Linux process

/// capabilities are defined in the Linux manpages (see the man7.org link below).

/// Default values can be found in: sources/models/shared-defaults/oci-capabilities.toml

/// OCI spec: /~https://github.com/opencontainers/runtime-spec/blob/c4ee7d12c742ffe806cd9350b6af3b4b19faed6f/config.md#linux-process

/// Linux kernel capabilities: https://man7.org/linux/man-pages/man7/capabilities.7.html

///

/// Default capabilities are specified in the following file:

/// * sources/models/shared-defaults/oci-capabilities.toml

/// OciDefaultsCapability specifies which process capabilities are

/// allowed to be set in the default OCI spec.

sources/models/shared-defaults/oci-capabilities.toml is extremely likely to be renamed during a refactor, and this comment probably won't stay correct. The OCI spec link seems a little out-of-place here, and the man page is pretty easy to find by anyone who knows they're working with Linux capabilities.

Agreed - wouldn't want stale references in comments. Accepted suggestion

bcressey · 2023-01-07T00:55:58Z

sources/models/src/modeled_types/oci_defaults.rs

+/// OciDefaultsResourceLimitType specifies which resource limits are
+/// allowed to be modified in Bottlerocket. The full list of Linux resource limits can
+/// be found in the Linux manpages (for convenience at the man7.org link below.)
+/// Default values can be found in: sources/models/shared-defaults/oci-resource-limits.toml
+/// OCI spec: /~https://github.com/opencontainers/runtime-spec/blob/c4ee7d12c742ffe806cd9350b6af3b4b19faed6f/config.md#posix-process
+/// Linux resource limits: https://man7.org/linux/man-pages/man2/getrlimit.2.html#DESCRIPTION
+///
+/// Default resource limits are specified in the following file:
+/// * sources/models/shared-defaults/oci-resource-limits.toml


Suggested change

/// OciDefaultsResourceLimitType specifies which resource limits are

/// allowed to be modified in Bottlerocket. The full list of Linux resource limits can

/// be found in the Linux manpages (for convenience at the man7.org link below.)

/// Default values can be found in: sources/models/shared-defaults/oci-resource-limits.toml

/// OCI spec: /~https://github.com/opencontainers/runtime-spec/blob/c4ee7d12c742ffe806cd9350b6af3b4b19faed6f/config.md#posix-process

/// Linux resource limits: https://man7.org/linux/man-pages/man2/getrlimit.2.html#DESCRIPTION

///

/// Default resource limits are specified in the following file:

/// * sources/models/shared-defaults/oci-resource-limits.toml

/// OciDefaultsResourceLimitType specifies which resource limits are

/// allowed to be set in the default OCI spec.

Accepted suggestion

mchaker · 2023-01-10T01:16:43Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/bf2b51bcb9b19109de89eceb060496a8b5c1f552..ec2a9f46efb263d54caee5b2df2f4d9be6c3fad2

Force push: address most PR comments

arnaldo2792 · 2023-01-10T01:48:50Z

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/src/main.rs

+    migrate(common_migrations::AddPrefixesMigration(vec![
+        "settings.oci-defaults",
+        "services.oci-defaults",
+        "configuration-files.oci-defaults",
+    ]))


Since this is a feature for k8s only, you may want to exclude the execution of the migration for all the other variants.

Done! Thanks :)

sources/Cargo.toml

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/src/main.rs

mchaker · 2023-01-11T17:45:12Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/66c11153cc3521157c6d03a93e049fa1e88af43a..e3d47c4f1be7227f08fa554b9c4aad6af025ae55

Force push: add metadata migration to Release.toml

etungsten

🍕

LGTM assuming all the testing looks good.

webern · 2023-01-13T19:35:02Z

^ rebase /~https://github.com/bottlerocket-os/bottlerocket/compare/e3d47c4f1be7227f08fa554b9c4aad6af025ae55..6a7d5a626706cf74155089a75f89db8170157bc7

webern · 2023-01-14T00:31:12Z

^ Attempt to fix metadata migration /~https://github.com/bottlerocket-os/bottlerocket/compare/6a7d5a626706cf74155089a75f89db8170157bc7..8be44e04b94087399ccd2a3d6c2957558f7711e5

webern · 2023-01-14T01:45:52Z

^ try even harder to fix the migrations /~https://github.com/bottlerocket-os/bottlerocket/compare/8be44e04b94087399ccd2a3d6c2957558f7711e5..8a04adde8601155ca1aa26389f7d522b94dde41a

stmcginnis

Looks good to me, testing done looks good, and tests are passing. I'm not strong on the migrations, so definitely would be good to have someone that understands those better take a closer look.

stmcginnis · 2023-01-16T08:58:38Z

sources/api/schnauzer/src/helpers.rs

+    let mut is_first = true;
+    for (rlimit_type, values) in &oci_default_rlimits {
+        if !is_first {
+            result_lines.push_str(",\n");


Just to note - I don't think it makes that much difference - but you could have results_line be an array, then use result_lines.join(",\n") at the end to get the resulting comma separated final string.

bcressey · 2023-01-17T17:12:01Z

Can you fix the co-authored-by tags in the commit messages?

Co-authored-by: name <mmchaker@amazon.com>
Co-authored-by: name <brigmatt@amazon.com>

Also, this description is not correct:

The default OCI spec stored at /etc/containerd/cri-base.json is applied to all containers run by containerd, unless otherwise configured (e.g. by Kubernetes pod specs).

The changes here only affect containerd-cri, and not containers launched by other means (host / bootstrap containers, Docker for ECS variants).

bcressey

LGTM apart from a few style nits and the commit messages.

bcressey · 2023-01-17T17:15:05Z

sources/api/migration/migrations/v1.12.0/oci-defaults-setting-metadata/src/main.rs

+    if cfg!(variant_runtime = "k8s") {
+        migrate(AddMetadataMigration(&[SettingMetadata {
+            metadata: &["affected-services"],
+            setting: "settings.oci-defaults",
+        }]))?;
+    };


If I understand correctly - conditionalizing this on the variant is just for defensive / documentation purposes, it isn't actually necessary for aws-ecs-1 which won't have this metadata.

bcressey · 2023-01-17T17:17:33Z

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/src/main.rs

+fn run() -> Result<()> {
+    if cfg!(variant_runtime = "k8s") {
+        migrate(common_migrations::AddPrefixesMigration(vec![
+            "settings.oci-defaults",
+            "services.oci-defaults",
+            "configuration-files.oci-defaults",
+        ]))
+    } else {
+        Ok(())
+    }
+}


to avoid the "else" branch:

Suggested change

fn run() -> Result<()> {

if cfg!(variant_runtime = "k8s") {

migrate(common_migrations::AddPrefixesMigration(vec![

"settings.oci-defaults",

"services.oci-defaults",

"configuration-files.oci-defaults",

]))

} else {

Ok(())

}

}

fn run() -> Result<()> {

if cfg!(variant_runtime = "k8s") {

migrate(common_migrations::AddPrefixesMigration(vec![

"settings.oci-defaults",

"services.oci-defaults",

"configuration-files.oci-defaults",

]))?;

}

Ok(())

}

bcressey · 2023-01-17T17:18:50Z

sources/api/migration/migrations/v1.12.0/oci-defaults-setting-metadata/src/main.rs

+    if cfg!(variant_runtime = "k8s") {
+        migrate(AddMetadataMigration(&[SettingMetadata {
+            metadata: &["affected-services"],
+            setting: "settings.oci-defaults",
+        }]))?;
+    };


nit: I don't think that last semicolon is needed:

Suggested change

if cfg!(variant_runtime = "k8s") {

migrate(AddMetadataMigration(&[SettingMetadata {

metadata: &["affected-services"],

setting: "settings.oci-defaults",

}]))?;

};

if cfg!(variant_runtime = "k8s") {

migrate(AddMetadataMigration(&[SettingMetadata {

metadata: &["affected-services"],

setting: "settings.oci-defaults",

}]))?;

}

bcressey · 2023-01-17T17:33:28Z

sources/api/schnauzer/src/helpers.rs

+    let capabilities_section = format!(
+        "\"bounding\": [
+{capabilities_bounding}
+],
+\"effective\": [
+{capabilities_effective}
+],
+\"permitted\": [
+{capabilities_permitted}
+]",
+        capabilities_bounding = capabilities_lines_joined,
+        capabilities_effective = capabilities_lines_joined,
+        capabilities_permitted = capabilities_lines_joined,
+    );


You can use a combination of raw strings and concat! to clean this up so the string literal isn't flush with the left margin:

Suggested change

let capabilities_section = format!(

"\"bounding\": [

{capabilities_bounding}

],

\"effective\": [

{capabilities_effective}

],

\"permitted\": [

{capabilities_permitted}

]",

capabilities_bounding = capabilities_lines_joined,

capabilities_effective = capabilities_lines_joined,

capabilities_permitted = capabilities_lines_joined,

);

let capabilities_section = format!(

concat!(

r#""bounding": ["#,

"{capabilities_bounding}",

"]",

r#""effective": ["#,

"{capabilities_effective}",

"]",

r#""permitted": ["#,

"{capabilities_permitted}",

"]",

),

capabilities_bounding = capabilities_lines_joined,

capabilities_effective = capabilities_lines_joined,

capabilities_permitted = capabilities_lines_joined,

);

(same for the rlimit function)

This is much nicer, thank you!

webern · 2023-01-17T19:17:39Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/8a04adde8601155ca1aa26389f7d522b94dde41a..684e813d7c88e27dba71a12c0c1bc7826110c141

Avoid the else branch
nit: I don't think that last semicolon is needed:
You can use a combination of raw strings and concat! to clean this up so the string literal isn't flush with the left margin:
Use Vec.join
Can you fix the co-authored-by tags in the commit messages?
The changes here only affect containerd-cri, and not containers launched by other means (host / bootstrap containers, Docker for ECS variants).

webern · 2023-01-17T21:25:37Z

^ /~https://github.com/bottlerocket-os/bottlerocket/compare/684e813d7c88e27dba71a12c0c1bc7826110c141..b0d09dc82fced3815b691c7ef14d5cd8f9cc3229

Fix a small oops

This change adds support for setting default OCI spec values. The default OCI spec stored at `/etc/containerd/cri-base.json` is applied to all containers run by `containerd-cri`, unless otherwise configured (e.g. by Kubernetes pod specs). Co-authored-by: Mahdi Chaker <mmchaker@amazon.com> Co-authored-by: Matthew James Briggs <brigmatt@amazon.com>

Co-authored-by: Mahdi Chaker <mmchaker@amazon.com> Co-authored-by: Matthew James Briggs <brigmatt@amazon.com>

webern · 2023-01-17T22:00:05Z

^ fix another oops /~https://github.com/bottlerocket-os/bottlerocket/compare/b0d09dc82fced3815b691c7ef14d5cd8f9cc3229..352a0a1477ff9084703c3be9ba8a0e1075ca1ce3

webern

I performed upgrade/downgrade testing from v1.11.0 -> v1.12.0 -> v1.11.0

On v1.11 and v1.12 I ran an ngnix pod and checked its oci spec with a command like this:

ctr -n k8s.io c info --spec 5a724a53cf

The specs were the same on both versions of Bottlerocket (i.e. if a customer does not use these new settings, their pods will not be affected).

👍

mchaker commented Jan 3, 2023

View reviewed changes

sources/models/shared-defaults/oci-capabilities.toml Outdated Show resolved Hide resolved

mchaker force-pushed the 1703-rlimits-clean branch from 98dc430 to 021d433 Compare January 3, 2023 23:56

mchaker force-pushed the 1703-rlimits-clean branch 2 times, most recently from 7dad2fb to 2a8101c Compare January 4, 2023 00:56

mchaker force-pushed the 1703-rlimits-clean branch 2 times, most recently from a3aed67 to afbec9e Compare January 4, 2023 21:34

webern reviewed Jan 5, 2023

View reviewed changes

sources/api/datastore/src/serialization/mod.rs Outdated Show resolved Hide resolved

mchaker force-pushed the 1703-rlimits-clean branch from afbec9e to 9c72669 Compare January 5, 2023 14:54

etungsten self-requested a review January 5, 2023 19:17

etungsten reviewed Jan 5, 2023

View reviewed changes

etungsten mentioned this pull request Jan 5, 2023

Add k8s 1.25 variants #2699

Merged

11 tasks

mchaker force-pushed the 1703-rlimits-clean branch from 9c72669 to e19d8eb Compare January 5, 2023 23:49

webern force-pushed the 1703-rlimits-clean branch from e19d8eb to c633a17 Compare January 6, 2023 00:19

webern force-pushed the 1703-rlimits-clean branch from c633a17 to 46d0035 Compare January 6, 2023 01:07

webern force-pushed the 1703-rlimits-clean branch from 46d0035 to 9d7001a Compare January 6, 2023 01:15

etungsten reviewed Jan 6, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

mchaker force-pushed the 1703-rlimits-clean branch from 9d7001a to bf2b51b Compare January 6, 2023 16:47

bcressey reviewed Jan 7, 2023

View reviewed changes

mchaker force-pushed the 1703-rlimits-clean branch from bf2b51b to ec2a9f4 Compare January 10, 2023 01:14

arnaldo2792 reviewed Jan 10, 2023

View reviewed changes

arnaldo2792 reviewed Jan 11, 2023

View reviewed changes

sources/Cargo.toml Show resolved Hide resolved

sources/api/migration/migrations/v1.12.0/oci-defaults-setting/src/main.rs Outdated Show resolved Hide resolved

mchaker force-pushed the 1703-rlimits-clean branch from 66c1115 to e3d47c4 Compare January 11, 2023 17:44

etungsten approved these changes Jan 13, 2023

View reviewed changes

webern force-pushed the 1703-rlimits-clean branch from e3d47c4 to 6a7d5a6 Compare January 13, 2023 19:33

webern force-pushed the 1703-rlimits-clean branch from 6a7d5a6 to 8be44e0 Compare January 14, 2023 00:30

webern force-pushed the 1703-rlimits-clean branch from 8be44e0 to 8a04add Compare January 14, 2023 01:45

stmcginnis approved these changes Jan 16, 2023

View reviewed changes

bcressey approved these changes Jan 17, 2023

View reviewed changes

webern force-pushed the 1703-rlimits-clean branch from 8a04add to 684e813 Compare January 17, 2023 19:16

webern force-pushed the 1703-rlimits-clean branch from 684e813 to b0d09dc Compare January 17, 2023 21:24

mchaker and others added 2 commits January 17, 2023 13:59

migrations: add migrations for 'settings.oci-defaults'

352a0a1

Co-authored-by: Mahdi Chaker <mmchaker@amazon.com> Co-authored-by: Matthew James Briggs <brigmatt@amazon.com>

webern force-pushed the 1703-rlimits-clean branch from b0d09dc to 352a0a1 Compare January 17, 2023 21:59

webern approved these changes Jan 17, 2023

View reviewed changes

mchaker merged commit 7b67f91 into bottlerocket-os:develop Jan 17, 2023

webern mentioned this pull request Jan 23, 2023

Update CHANGELOG.md for v1.12.0 #2746

Merged

mchaker mentioned this pull request Jan 30, 2023

Allow overrides to the default OCI spec capabilities for k8s pods #2404

Closed

webern mentioned this pull request Jan 30, 2023

support settings.oci-defaults on ecs variants #2759

Closed

webern mentioned this pull request Feb 16, 2023

Upstream cri patch for default RLIMIT_NOFILE #1240

Closed

webern mentioned this pull request Apr 6, 2023

Additional kubelet configuration settings #2930

Merged

vyaghras mentioned this pull request Jul 12, 2023

Support settings oci defaults on ecs variants #3259

Merged

8 tasks

bcressey mentioned this pull request Jan 4, 2024

containerd, docker: consider removing LimitNOFILE specification from service units. #3638

Closed


		The default capabilities that are set in Bottlerocket are:

		setting key \| setting value

		[metadata.settings.oci-defaults]
		affected-services = ["oci-defaults", "containerd"]

		@@ -0,0 +1 @@
		../../../shared-defaults/oci-defaults.toml

containerd: add OCI default spec and settings #2697

containerd: add OCI default spec and settings #2697

Conversation

mchaker commented Jan 3, 2023 • edited by webern Loading

Testing done:

OCI Defaults Test Plan

Manual Testing

Baseline

Test case: feature enabled and configured with default settings

Test case: set capabilities

Test case: set resource-limits

Test case: set both capabilities and resource-limits

Test all variants

Run Sonobuoy Conformance

Test case: Upgrade Downgrade

mchaker left a comment • edited Loading

Choose a reason for hiding this comment

mchaker commented Jan 3, 2023

mchaker commented Jan 4, 2023

mchaker commented Jan 5, 2023

mchaker commented Jan 5, 2023

webern commented Jan 6, 2023

webern commented Jan 6, 2023

webern commented Jan 6, 2023

mchaker commented Jan 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mchaker commented Jan 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mchaker commented Jan 11, 2023

etungsten left a comment • edited Loading

Choose a reason for hiding this comment

webern commented Jan 13, 2023

webern commented Jan 14, 2023

webern commented Jan 14, 2023

stmcginnis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bcressey commented Jan 17, 2023

bcressey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webern commented Jan 17, 2023

webern commented Jan 17, 2023

webern commented Jan 17, 2023

webern left a comment

Choose a reason for hiding this comment

mchaker commented Jan 3, 2023 •

edited by webern

Loading

mchaker left a comment •

edited

Loading

etungsten left a comment •

edited

Loading