Shared folder mounts randomly get unmounted #20398

ncdc · 2023-10-18T14:07:35Z

Issue Description

I'm running Podman on a Mac, and the Podman machine VM randomly loses its shared folder mounts. Here's an example where my VM had been up for a while, and I tried to run a container that uses a bind mount from the "host" (my Mac) and it failed:

make jobs prow-config
false || podman pull --platform linux/amd64 registry.ci.openshift.org/ci/ci-operator-checkconfig:latest
Trying to pull registry.ci.openshift.org/ci/ci-operator-checkconfig:latest...
Getting image source signatures
Copying blob sha256:09926637c7bf9825d7afb25218a2b16128eea30d547297bd73d6e53c07025396
Copying blob sha256:8ed15414cb45b8b895a21f106743ceee45272cee0d9e7e5b1f49bd1e877e4a11
Copying config sha256:0cfba2e738cf9bbc204bda5d586d0ed4ae8dfae05826f79b6b140e5d94f91fea
Writing manifest to image destination
0cfba2e738cf9bbc204bda5d586d0ed4ae8dfae05826f79b6b140e5d94f91fea
podman run --platform linux/amd64  --rm -v "/Users/angoldst/code/openshift/release/ci-operator/config:/ci-operator/config" -v "/Users/angoldst/code/openshift/release/ci-operator/step-registry:/ci-operator/step-registry" registry.ci.openshift.org/ci/ci-operator-checkconfig:latest --config-dir /ci-operator/config --registry /ci-operator/step-registry
Error: statfs /Users/angoldst/code/openshift/release/ci-operator/config: no such file or directory
make: *** [ci-operator-checkconfig] Error 125

Logging in to the VM indeed shows the mount is missing:

podman machine ssh 
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 38.20230918.3.2
Tracker: /~https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Tue Oct 17 12:04:33 2023 from 192.168.127.1
[core@localhost ~]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
devtmpfs            4096        0      4096   0% /dev
tmpfs            4058768      168   4058600   1% /dev/shm
tmpfs            1623508     8880   1614628   1% /run
/dev/vda4      104266732 14156312  90110420  14% /sysroot
tmpfs            4058772        0   4058772   0% /tmp
/dev/vda3         358271   206468    128047  62% /boot
tmpfs             811752      164    811588   1% /run/user/501

Restarting the VM results in the missing mount returning:

podman machine stop && podman machine start
Waiting for VM to exit...
Machine "podman-machine-default" stopped successfully
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

        podman machine set --rootful

API forwarding listening on: /Users/angoldst/.local/share/containers/podman/machine/qemu/podman.sock

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the\nfollowing commands:

        sudo /usr/local/Cellar/podman/4.7.1/bin/podman-mac-helper install
        podman machine stop/usr/local/Cellar/podman/4.7.1/bin/podman-mac-helper; podman machine start/usr/local/Cellar/podman/4.7.1/bin/podman-mac-helper

                You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

        export DOCKER_HOST='unix:///Users/angoldst/.local/share/containers/podman/machine/qemu/podman.sock'

Machine "podman-machine-default" started successfully
(⎈|default/api-ci-l2s4-p1-openshiftapps-com:6443/angoldst:default) angoldst@angoldst-mac  ~/code/openshift/release   rukpak-z-e2e ±  podman machine ssh                          
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 38.20230918.3.2
Tracker: /~https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Wed Oct 18 09:56:58 2023 from 192.168.127.1
df[core@localhost ~]$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs            4096         0      4096   0% /dev
tmpfs            4058768        84   4058684   1% /dev/shm
tmpfs            1623508      8888   1614620   1% /run
/dev/vda4      104266732  14221980  90044752  14% /sysroot
tmpfs            4058772         0   4058772   0% /tmp
/dev/vda3         358271    206468    128047  62% /boot
tmpfs             811752         8    811744   1% /run/user/501
vol0           488245288 415811556  72433732  86% /Users
vol1           488245288 415811556  72433732  86% /private
vol2           488245288 415811556  72433732  86% /var/folders

In this case, it's vol0.

Steps to reproduce the issue

podman machine start
wait some amount of time (?)

Describe the results you received

Shared folder mount disappears

Describe the results you expected

Not that :smi

podman info output

Here it is from the VM:


host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.35
    systemPercent: 0.45
    userPercent: 0.2
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7885225984
  memTotal: 8312360960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 7m 10.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 14563307520
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 105
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549242
  BuiltTime: Tue Sep 12 16:07:22 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Here it is from my Mac:

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.41
    systemPercent: 0.41
    userPercent: 0.18
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7887257600
  memTotal: 8312360960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 8m 0.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 14563307520
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 105
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549242
  BuiltTime: Tue Sep 12 16:07:22 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2



### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

Using the stable ostree `rpm-ostree rebase fedora/x86_64/coreos/stable`

### Additional information

_No response_

The text was updated successfully, but these errors were encountered:

Luap99 · 2023-10-18T15:59:17Z

see #20122 and #15976

I assume your VM rebooted automatically due an upgrade. Given the above this should no longer be the case with VM's created with future versions.

ncdc added the kind/bug Categorizes issue or PR as related to a bug. label Oct 18, 2023

github-actions bot added the remote Problem is in podman-remote label Oct 18, 2023

Luap99 added the machine label Oct 18, 2023

Luap99 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 18, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jan 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shared folder mounts randomly get unmounted #20398

Shared folder mounts randomly get unmounted #20398

ncdc commented Oct 18, 2023

Luap99 commented Oct 18, 2023

Shared folder mounts randomly get unmounted #20398

Shared folder mounts randomly get unmounted #20398

Comments

ncdc commented Oct 18, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Luap99 commented Oct 18, 2023