Support for Linux Kernel 5.1+

[cjheppell]

Is it possible to support Linux kernel 5.1+ in the built Minikube iso?

The reason for this is that I'm intending to run Ceph in Minikube (for which the ceph-csi plugin requires kernel 5.1+ or higher for "deep flatten" support).

It looks like this was done in #8187, but then reverted later in #8649.

I've tried to manually rebuild the Minikube iso myself by effectively reinstating the changes in #8187, but I've ended up with a Virtualbox VM that doesn't start. 😞

[afbjorklund]

This is planned for next major upgrade of the OS in the VM...

It would be either of 5.4 or 5.10, which are both 5.1+ I guess.

Just have to decide if it is Buildroot 2021.02 or Ubuntu 20.04 :-)

See #9992

[afbjorklund]

Here are the BR2_LINUX_KERNEL_LATEST_VERSION, for the LTS releases:

Buildroot	Kernel
2018.02.x	4.15
2019.02.x	4.19
2020.02.x	5.4
2021.02.x	5.10

See https://buildroot.org/news.html for the release announcements and details.

As you noticed, we downgraded to an earlier kernel because of issues in testing...

[cjheppell]

Great stuff. Thanks for sharing the links.

I'm very keen to see this happen, as we could really do with a 5.1+ kernel.

Is there a preexisting iso from the version that was targeting the 5.4 kernel from the PRs I mentioned?

It's a very significant blocker for me, so I'd be keen to use an older, "unsupported" iso if its based on 5.1+ until the official support is there?

[ilya-zuyev]

@cjheppell unfortunately, we don't keep ISOs for PRs. It's in our plans.
You can try to build an ISO locally using PR branch, but it's almost a year old and can be outdated a bit.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[cjheppell]

/remove-lifecycle stale

[fredjeronimo]

Hi all. Is there an ETA by any chance for this upgrade?

We are also using Minikube with Ceph in our development cycle, so very keen to get this in if possible.

The main goal is to avoid using our Azure testing cluster for our developers' local testing but, at present, the flatten in progress that occurs when restoring snapshots from rbd images with deep-flatten image feature is very time consuming and sadly makes Minikube impractical in our use case.

[sharifelgamal]

We definitely want to do this, but our past couple of attempts have resulted in failures that we haven't had the bandwidth to address them. If anyone wants to give this a shot, we would absolutely accept a PR.

You can test building an ISO locally (on linux) by running make minikube_iso, with the caveat that it takes about an hour.

[eiffel-fl]

Hi.

I also would like to be able to use a more recent kernel version than what is actually provided by minikube.
For now, I was able to boot the ISO by adding the following modifications:

diff --git a/Makefile b/Makefile
index 42bdb7131..241533317 100644
--- a/Makefile
+++ b/Makefile
@@ -73,7 +73,7 @@ MINIKUBE_BUCKET ?= minikube/releases
 MINIKUBE_UPLOAD_LOCATION := gs://${MINIKUBE_BUCKET}
 MINIKUBE_RELEASES_URL=/~https://github.com/kubernetes/minikube/releases/download
 
-KERNEL_VERSION ?= 4.19.202
+KERNEL_VERSION ?= 5.10.57
 # latest from /~https://github.com/golangci/golangci-lint/releases 
 # update this only by running `make update-golint-version`
 GOLINT_VERSION ?= v1.42.1
diff --git a/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig b/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
index 1871f86d8..4b31f4513 100644
--- a/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
+++ b/deploy/iso/minikube-iso/board/coreos/minikube/linux_defconfig
@@ -1,3 +1,4 @@
+CONFIG_TMPFS=y
 # CONFIG_LOCALVERSION_AUTO is not set
 CONFIG_KERNEL_LZ4=y
 CONFIG_SYSVIPC=y
diff --git a/deploy/iso/minikube-iso/configs/minikube_defconfig b/deploy/iso/minikube-iso/configs/minikube_defconfig
index 9cd682c72..55dbabba4 100644
--- a/deploy/iso/minikube-iso/configs/minikube_defconfig
+++ b/deploy/iso/minikube-iso/configs/minikube_defconfig
@@ -4,7 +4,7 @@ BR2_OPTIMIZE_2=y
 BR2_GLOBAL_PATCH_DIR="$(BR2_EXTERNAL_MINIKUBE_PATH)/board/coreos/minikube/patches"
 BR2_TOOLCHAIN_BUILDROOT_VENDOR="minikube"
 BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
-BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_4_19=y
+BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_10=y
 BR2_TOOLCHAIN_BUILDROOT_CXX=y
 BR2_GCC_ENABLE_LTO=y
 BR2_TARGET_GENERIC_HOSTNAME="minikube"
@@ -17,8 +17,8 @@ BR2_SYSTEM_BIN_SH_BASH=y
 BR2_ROOTFS_USERS_TABLES="$(BR2_EXTERNAL_MINIKUBE_PATH)/board/coreos/minikube/users"
 BR2_ROOTFS_OVERLAY="$(BR2_EXTERNAL_MINIKUBE_PATH)/board/coreos/minikube/rootfs-overlay"
 BR2_LINUX_KERNEL=y
-BR2_LINUX_KERNEL_CUSTOM_VERSION=y
-BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.19.202"
+BR2_LINUX_KERNEL_LATEST_VERSION=y
+BR2_LINUX_KERNEL_VERSION="5.10.57"
 BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
 BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="$(BR2_EXTERNAL_MINIKUBE_PATH)/board/coreos/minikube/linux_defconfig"
 BR2_LINUX_KERNEL_LZ4=y
diff --git a/deploy/iso/minikube-iso/package/hyperv-daemons/hyperv-daemons.mk b/deploy/iso/minikube-iso/package/hyperv-daemons/hyperv-daemons.mk
index 3ea36bbbb..dcc26010e 100644
--- a/deploy/iso/minikube-iso/package/hyperv-daemons/hyperv-daemons.mk
+++ b/deploy/iso/minikube-iso/package/hyperv-daemons/hyperv-daemons.mk
@@ -5,7 +5,7 @@
 ################################################################################
 
 HYPERV_DAEMONS_VERSION = $(call qstrip,$(BR2_LINUX_KERNEL_VERSION))
-HYPERV_DAEMONS_SITE = https://www.kernel.org/pub/linux/kernel/v4.x
+HYPERV_DAEMONS_SITE = https://www.kernel.org/pub/linux/kernel/v5.x
 HYPERV_DAEMONS_SOURCE = linux-$(HYPERV_DAEMONS_VERSION).tar.xz
 
 define HYPERV_DAEMONS_BUILD_CMDS

I can boot it with qemu-system-x86_64 -enable-kvm -nographic -m 2G -cdrom out/minikube.iso but I am getting some service errors:

[FAILED] Failed to listen on Docker Socket for the API.
See 'systemctl status docker.socket' for details.
         Starting Podman API Socket.
         Starting minikube automount...
[FAILED] Failed to start minikube scheduled stop.
See 'systemctl status minikube-scheduled-stop.service' for details.
[  OK  ] Finished minikube automount.
[FAILED] Failed to start CRI-O Auto Update Script.
See 'systemctl status crio-wipe.service' for details.
[DEPEND] Dependency failed for Cont…ime Interface for OCI (CRI-O).

And I cannot get it to work through minikube start:

/ I1013 15:25:04.958238  289919 main.go:130] libmachine: (minikube) DBG | domain minikube has defined MAC address 52:54:00:fe:32:42 in network me
I1013 15:25:04.958727  289919 main.go:130] libmachine: (minikube) DBG | unable to find current IP address of domain minikube in network mk-minike
I1013 15:25:04.958798  289919 main.go:130] libmachine: (minikube) DBG | I1013 15:25:04.958704  289940 retry.go:31] will retry after 4.276119362sp

If I can dedicate a bit of time, I think I should be able to solve the ones related to services.
For the one in minikube start, I lack some knowledge to solve it.

Best regards.

[sharifelgamal]

Bumping up the priority since there should be fewer blockers to getting this done.

[kakkoyun]

It would be amazing to have support for 5.2+ to use BTF with eBPF based programs #8556 (comment)

[medyagh]

related #12707

[afbjorklund]

Scope this out of the ISO until we have a working updated image for arm64/aarch64.

[afbjorklund]

Here are the BR2_LINUX_KERNEL_LATEST_VERSION, for the LTS releases:

Buildroot	Kernel
2018.02.x	4.15
2019.02.x	4.19
2020.02.x	5.4
2021.02.x	5.10
2022.02.x	5.15

[medyagh]

I think this is done and we could close after verification

[spowelljr]

Confirmed:

$ minikube version
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565
$ minikube ssh
$ uname -r
5.10.57