Mount static kmod as /sbin/modprobe

[vigh-m]

Issue number:

Closes #3968

Description of changes:
This change mounts the static kmod built in #3981 into a customer container.

Testing done:
Tested on the host-ctr:

[ssm-user@control]$ enter-admin-container
Confirming admin container is enabled...
Waiting for admin container to start........
Entering admin container
          Welcome to Bottlerocket's admin container!
    ╱╲
   ╱┄┄╲   This container provides access to the Bottlerocket host
   │▗▖│   filesystems (see /.bottlerocket/rootfs) and contains common
  ╱│  │╲  tools for inspection and troubleshooting.  It is based on
  │╰╮╭╯│  Amazon Linux 2, and most things are in the same places you
    ╹╹    would find them on an AL2 host.

To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`).  When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
[root@admin]# ldd /usr/sbin/modprobe
  not a dynamic executable

Tested on a kubernetes container:

root@my-pod:/# ldd /usr/sbin/modprobe
  not a dynamic executable
root@my-pod:/# ldd /sbin/modprobe
  not a dynamic executable
root@my-pod:/# ldd /usr/bin/kmod
ldd: /usr/bin/kmod: No such file or directory

Pod Spec used:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: ubuntu:noble
    command: ["/bin/sh", "-c", "while true; do echo 'Running...'; sleep 5; done"]
    securityContext:
      capabilities:
        add: ["SYS_MODULE"]
    volumeMounts:
    - name: kernel-modules
      mountPath: /lib/modules/
      readOnly: true
  volumes:
  - name: kernel-modules
    hostPath:
      path: /lib/modules/

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[bcressey]

Code changes LGTM, though I would split them into one for host-ctr and one for containerd.

Does it work for Cilium? For the example pod spec, can you load a module via modprobe?

[yeazelm]

The code looks good, it would be awesome to include a before and after on that pod test to show the change, even better showing it loading a module after the fix to prove this fixes the issue.

[vigh-m]

Yeah. It loads the modules like expected.

root@my-pod:/# ln -s /usr/sbin/modprobe /usr/bin/lsmod
root@my-pod:/# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables
root@my-pod:/# modprobe -r nf_tables
root@my-pod:/# lsmod | grep tables
root@my-pod:/# modprobe nf_tables
root@my-pod:/# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables

I had validated but didn't add it to the initial notes

[vigh-m]

Validated via SHA512 that it's the same modprobe

• On admin container

[root@admin]# sha512sum /usr/bin/kmod
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /usr/bin/kmod

• On rootfs

[root@admin]# sha512sum /.bottlerocket/rootfs/bin/kmod
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /.bottlerocket/rootfs/bin/kmod

• On container

root@my-pod:/# sha512sum /usr/sbin/modprobe
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /usr/sbin/modprobe

[arnaldo2792]

For the validation, you could just have checked that the modprobe binary is coming from the host via /proc/self/mountinfo.

[arnaldo2792]

One risk of this implementation is that if PATH is /usr/bin:/sbin, the modprobe binary in the container will be used instead of the injected modprobe. I guess that's a calculated risk to take, since we can't inject modprobe in every single path in PATH. Probably something to follow up for a later release is checking PATH and inject modprobe to the first path in that list.

[bcressey]

It would be good to validate that Cilium specifically works as expected with this change, in case there are more paths we need to bind to.

[vigh-m]

New changes include:

• Updates to docker to inject kmod by default
• Fix based on comments
• Typo fixes

[vigh-m]

Fixing typo in the commit message

[vigh-m]

Validated by running the cilium/cilium:v1.15.5 image from DockerHub:

Locally launched container:

$ docker run -it --rm cilium/cilium:v1.15.5 sh
# ldd /usr/sbin/modprobe
	linux-vdso.so.1 (0x00007fffc838c000)
	libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f6632d16000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f6632ceb000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f66328a7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f663267e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6632e14000)
# ls -la /usr/sbin/modprobe
lrwxrwxrwx. 1 root root 9 Aug 17  2021 /usr/sbin/modprobe -> /bin/kmod
# cat /proc/self/mountinfo  | grep kmod
#

VS container launched on bottlerocket

$ kubectl exec -it my-pod -- bash
root@my-pod:/home/cilium#
root@my-pod:/home/cilium# ldd /usr/sbin/modprobe
	not a dynamic executable
root@my-pod:/home/cilium# ls -la /usr/sbin/modprobe
lrwxrwxrwx. 1 root root 9 Aug 17  2021 /usr/sbin/modprobe -> /bin/kmod
root@my-pod:/home/cilium# ldd /bin/kmod
	not a dynamic executable
root@my-pod:/home/cilium# cat /proc/self/mountinfo | grep kmod
1281 1275 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/kmod /usr/bin/kmod ro,relatime master:1 - ext4 /dev/root ro,seclabel,stripe=1024

I'm able to load and unload modules

root@my-pod:/home/cilium# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables
root@my-pod:/home/cilium# modprobe -r nf_tables
root@my-pod:/home/cilium# lsmod | grep tables
root@my-pod:/home/cilium# modprobe nf_tables
root@my-pod:/home/cilium# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables