Unable to build a devcontainer features with podman

[dubielt1]

.devcontainer/Dockerfile:

FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04

RUN apt-get update && apt-get install -y \
    foot-terminfo \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /opt/
 RUN wget /~https://github.com/neovim/neovim/releases/download/stable/nvim-linux64.tar.gz \
    && tar xzvf ./nvim-linux64.tar.gz \
    && rm ./nvim-linux64.tar.gz

ENV PATH="$PATH:/opt/nvim-linux64/bin"

.devcontainer/devcontainer.json:

{
  "name": "ide-py",
  "build": {
    "dockerfile": "Dockerfile"
  },
  "features": {
    "ghcr.io/devcontainers/features/python:1": {
      "version": "latest"
    }
  }
  //"runArgs": ["--security-opt", "label=disable"],
}

build command: devcontainer build --docker-path podman --workspace-folder ./

devcontainer Error Encountered

[3/3] STEP 10/13: RUN --mount=type=bind,from=dev_containers_feature_content_source,source=python_1,target=/tmp/build-features-src/python_1     cp -ar /tmp/build-features-src/python_1 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/python_1  && cd /tmp/dev-container-features/python_1  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/python_1
cp: cannot access '/tmp/build-features-src/python_1': Permission denied
Error: building at STEP "RUN --mount=type=bind,from=dev_containers_feature_content_source,source=python_1,target=/tmp/build-features-src/python_1 cp -ar /tmp/build-features-src/python_1 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/python_1  && cd /tmp/dev-container-features/python_1  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/python_1": while running runtime: exit status 1
Error: Command failed: podman buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-tdubiel/container-features/0.43.0-1686449407786/Dockerfile-with-features -t vsc-py-28d0486e1c2b00c4e973d123ef6f69141ddf0a260b975d4df1c9e87959307d37 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-tdubiel/container-features/0.43.0-1686449407786 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/home/tdubiel/Code/ide-toolbox-setup/py/.devcontainer
  at xte (/var/home/tdubiel/.nvm/versions/node/v18.16.0/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:2005:1698)
  at async zw (/var/home/tdubiel/.nvm/versions/node/v18.16.0/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:2004:1691)
  at async cne (/var/home/tdubiel/.nvm/versions/node/v18.16.0/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:2152:20389)
  at async lne (/var/home/tdubiel/.nvm/versions/node/v18.16.0/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:2152:18322)
{"outcome":"error","message":"Command failed: podman buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-tdubiel/container-features/0.43.0-1686449407786/Dockerfile-with-features -t vsc-py-28d0486e1c2b00c4e973d123ef6f69141ddf0a260b975d4df1c9e87959307d37 --target dev_containers_target_stage --build-context dev_containers_feature_content_source=/tmp/devcontainercli-tdubiel/container-features/0.43.0-1686449407786 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /var/home/tdubiel/Code/ide-toolbox-setup/py/.devcontainer","description":"An error occurred building the image."}

podman info --debug output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 97.24
    systemPercent: 1.76
    userPercent: 0.99
  cpus: 16
  distribution:
    distribution: fedora
    variant: sericea
    version: "38"
  eventLogger: journald
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.2.9-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 22976339968
  memTotal: 33416204288
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.3-2.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 2h 30m 42.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/tdubiel/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 3
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tdubiel/.local/share/containers/storage
  graphRootAllocated: 137338290176
  graphRootUsed: 8593375232
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 48
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/tdubiel/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.2
  Built: 1677669759
  BuiltTime: Wed Mar  1 05:22:39 2023
  GitCommit: ""
  GoVersion: go1.20.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.2

env:

• node --version: v18.16.0
• devcontainer --version: 0.43.0

---

Possibly related topics?

• SELinux shared labels: Build with --mount=type=bind can't access the mounted folder containers/podman#15423

[fsdrw08]

some to me, I think podman does not support podman buildx command

[chrmarti]

Related: microsoft/vscode-remote-release#1333

[dubielt1]

@chrmarti - thank you sharing the related ticket

I must admit, I am not sure which rabbit hole to start with, or if the implication is that this won't work at all... an additional hint if this is possible would be appreciated

---

I created a $HOME/.config/containers/storage.conf, with these contents: (docs for future readers)

[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

followed by podman system reset and a re-attempted devcontainer build..., but it failed in the same manner as before.

---

I then followed the above procedure, but for $HOME/.config/containers/containers.conf: (docs for future readers)

[containers]
volumes=["/tmp:/tmp:Z"]

This also failed, much earlier in the build process. This attempt was probably misguided.

---

If permissive/disabled SELinux is a viable solution, I would be open to that. This is my system for personal dev only

[chrmarti]

I'm not familiar with SELinux. Commenters in microsoft/vscode-remote-release#1333 have suggested that the z/Z flags (https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label) might help, but I haven't tried. Note that the documentation warns about using Z on system folders.

For the temporary mount in the Dockerfile we could look into using these flags although it is unclear if they are supported (the command line --mount option does not support them, only -v does).

If you can disable the filesystem level protection of your SELinux setup, maybe give that a try. (Not sure if that is possible at all.)

[dubielt1]

@chrmarti, after looking into what the z/Z flags do, I found a way to do the same thing with SELinux commands. The below successfully builds the devcontainer, without permanently changing the system or SELinux config.

chcon tweaks SELinux so that the build succeeds, and restorecon reverts said-tweak

sudo chcon -R -t container_file_t /tmp/
devcontainer build --docker-path podman --workspace-folder ./
sudo restorecon -RFv /tmp/

Disclaimer: I lack the expertise to determine if there are downsides to doing this. It seems harmless

Notes for the curious:

• The z flag does a similar thing to the above command, relabeling the directory as container_file_t (src)
• restorecon required the F (force) flag. Without it, it would log the following when using the verbose flag: /tmp not reset as customized by admin to system_u:object_r:container_file_t:s0
• If future readers want to explore setting /tmp as container_file_t permanently, don't use chcon. Search for semanage fcontext

[gilesknap]

@dubielt1 please consider re-opening this bug.

Making changes to the security settings on /tmp is not available to non-root users. At our facility we don't give root access to developers, only the sysadmins have this right.

podman fits into this approach perfectly because it allows us to build and use containers without root permission.

I've already asked our sysadmins to change the SELinux settings on /tmp on our workstation build. Their response was - that is the default settings that RedHat provide so we don't want to change them.

I assume at present devcontainers do not change the command line arguments they use based upon podman/docker. But really in order to support podman on its native distro of RedHat this needs to be the case. Or at least expose the arguments as a setting that we can override.

In the devcontainer JSON we get to specify the arguments for building containers. But the cause of this issue (which happens when adding 'features' to your devcontainer) is not exposed in the same way.

[chrmarti]

@gilesknap I have added a fix that will use the z flag for this mount when using buildah. Hope that makes sense. This change will be in the next version of the CLI.

[gilesknap]

@chrmarti thanks! Sorry, I missed the mention of the commit. That is exactly the fix we were hoping for.

[worldofgeese]

@gilesknap I have added a fix that will use the z flag for this mount when using buildah. Hope that makes sense. This change will be in the next version of the CLI.

@chrmarti, @n1hility of Red Hat was taking a look at this issue too. Does this fix solve for all rootless use-cases of Podman powering devcontainers?

[chrmarti]

@worldofgeese This only takes care of the build context for adding feature scripts.