airgap k3s v1.30.4+k3s1: FATA[0000] exec: "k3s-agent": executable file not found in $PATH

[cwrx777]

Environmental Info:
K3s Version:

k3s version v1.30.4+k3s1 (98262b5d)

Node(s) CPU architecture, OS, and Version:
Linux 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 1 04:16:12 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server, 1 agent

Describe the bug:
Failed to start agent using airgap image (k3s-airgap-images-amd64.tar.gz and k3s-airgap-images-amd64.tar)

FATA[0000] exec: "k3s-agent": executable file not found in $PATH

No issue with v1.29.8+k3s1 airgap images.

Steps To Reproduce:
Copy airgap image to /var/lib/rancher/k3s/agent/images/
Copy k3s binary to /usr/local/bin/
Start agent:

sudo systemctl start k3s-agent.service

• Installed K3s: v1.30+k3s1

Additional context / logs:

$ sudo ls -lart /var/lib/rancher/k3s/data
total 4
-rw-------. 1 root root    0 Sep 13 10:29 .lock
drwx------. 5 root root   45 Sep 13 10:29 ..
drwxr-xr-x. 4 root root   28 Sep 13 10:29 c38ba7cc1669e7d80b8156ae743932fd86f5bce3871b8a88bef531dd4e3c02b2
lrwxrwxrwx. 1 root root   90 Sep 13 10:30 previous -> /var/lib/rancher/k3s/data/c38ba7cc1669e7d80b8156ae743932fd86f5bce3871b8a88bef531dd4e3c02b2
drw-------. 4 root root   28 Sep 13 10:34 e50868881d9744d0d0027dda983507e867b3787482eb00005d97239d9aa501a5
lrwxrwxrwx. 1 root root   90 Sep 13 10:34 current -> /var/lib/rancher/k3s/data/e50868881d9744d0d0027dda983507e867b3787482eb00005d97239d9aa501a5
drwxr-xr-x. 4 root root 4096 Sep 13 10:34 .


$ sudo ls -lart /var/lib/rancher/k3s/data/e50868881d9744d0d0027dda983507e867b3787482eb00005d97239d9aa501a5/bin  | grep k3s
-rw-------. 1 root root 174180208 Aug 22 07:06 k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 kubectl -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 containerd -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 crictl -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-agent -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-secrets-encrypt -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-server -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-etcd-snapshot -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-certificate -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-completion -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 k3s-token -> k3s
lrwxrwxrwx. 1 root root         3 Sep 13 10:34 ctr -> k3s

sudo ls -lart /var/lib/rancher/k3s/data/current/bin
total 202424
-rw-------. 1 root root     96000 Jul 10 03:24  xargs
-rw-------. 1 root root   1121192 Jul 10 03:24  slirp4netns
-rw-------. 1 root root    177944 Jul 10 03:24  pigz
-rw-------. 1 root root     71328 Jul 10 03:24  nsenter
-rw-------. 1 root root    198144 Jul 10 03:24  losetup
-rw-------. 1 root root    204320 Jul 10 03:24  ipset
-rw-------. 1 root root    598304 Jul 10 03:24  ip
-rw-------. 1 root root    444536 Jul 10 03:24  fuse-overlayfs
-rw-------. 1 root root    289712 Jul 10 03:24  find
-rw-------. 1 root root    243992 Jul 10 03:24  ethtool
-rw-------. 1 root root   1311008 Jul 10 03:24  coreutils
-rw-------. 1 root root    195952 Jul 10 03:24  conntrack
-rw-------. 1 root root    812720 Jul 10 03:24  busybox
-rw-------. 1 root root    305472 Jul 10 03:24  blkid
-rw-------. 1 root root   4325528 Aug 22 07:05  cni
-rw-------. 1 root root 174180208 Aug 22 07:06  k3s
-rw-------. 1 root root  12484760 Aug 22 07:06  containerd-shim-runc-v2
-rw-------. 1 root root      2103 Aug 22 07:06  .sha256sums
-rw-------. 1 root root  10137480 Aug 22 07:06  runc
-rw-------. 1 root root      6143 Aug 22 07:06  .links
-rw-------. 1 root root     14141 Aug 22 07:06  check-config

[cwrx777]

After changing file permission in /var/lib/rancher/k3s/data/current/bin, it is able to start.

 sudo chmod ugo+rx -R /var/lib/rancher/k3s/data/current/bin

[brandond]

Something prevented the extract process from completing successfully. This has nothing to do with airgap, the content in question is embedded in the k3s binary itself. Did you perhaps interrupt the process while it was starting?

[cwrx777]

Something prevented the extract process from completing successfully. This has nothing to do with airgap, the content in question is embedded in the k3s binary itself. Did you perhaps interrupt the process while it was starting?

I tried in another set of VMs and no issue in installing the agent for v1.30.4+k3s1.