Container Can not Run: oci runtime error: container_linux.go: not a directory. When mounting Configmaps and a subpath in the same directory

[Joseph-Irving]

/kind bug

What happened:
After upgrading to 1.94 (now on 1.9.6) I could not run my flannel container

The error is

invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:53: mounting \\\\\\\"/var/lib/kubelet/pods/d1d414bd-2db0-11e8-a1f1-024f81364208/volume-subpaths/flannel-net-conf/kube-flannel/2\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/9a9befa0846d036e5da22e8bb9eaca6ce313aed97671629a96dc79323df4127c/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/9a9befa0846d036e5da22e8bb9eaca6ce313aed97671629a96dc79323df4127c/merged/etc/kube-flannel/net-conf.json\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n"

What you expected to happen:
The container would be created and run
How to reproduce it (as minimally and precisely as possible):
Using this deployment I could reproduce it, the important part being that I'm mounting the configmap as directory and then mounting a file from the emptyDir as a SubPath in the same directory.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: subpath
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: subpath
    spec:
      initContainers:
        - name: init
          image: busybox
          command:
            - touch
            - /mount/test
          volumeMounts:
            - name: mount
              mountPath: /mount
      containers:
      - name: subtest
        image: busybox
        command:
          - sleep
          - "300"
        volumeMounts:
          - name: mount
            mountPath: /mount/test
            subPath: test
          - name: config
            mountPath: /mount
      volumes:
      - name: mount
        emptyDir: {}
      - name: config
        configMap:
          name: mount
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mount
  namespace: kube-system
data:
  hello.txt: |
    hello

Anything else we need to know?:

Environment::

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6+coreos.0", GitCommit:"c2aac694c2c7373dfe3ad81f47fc3cbc70a5a8fa", GitTreeState:"clean", BuildDate:"2018-03-21T21:54:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  AWS
• OS (e.g. from /etc/os-release):

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.2.1
VERSION_ID=1632.2.1
BUILD_ID=2018-02-01-2053
PRETTY_NAME="Container Linux by CoreOS 1632.2.1 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Kernel (e.g. uname -a):

Linux ip-172-23-10-171 4.14.16-coreos #1 SMP Thu Feb 1 20:38:35 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux

• Install tools:
• Others:
  Docker:

docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.6
 Git commit:   d9ad3fc
 Built:        Thu Feb  1 20:40:03 2018
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.6
 Git commit:   d9ad3fc
 Built:        Thu Feb  1 20:40:03 2018
 OS/Arch:      linux/amd64

Storage driver is overlay2

[liggitt]

/sig storage
/assign @msau42 @rootfs @joelsmith

joel, does this look related to the overlapping volume work you did around the atomic writer volumes? (#60342)

[liggitt]

/kind bug

[liggitt]

cc @jsafrane

[joelsmith]

A limitation of #60342 is that it assumes that mount points are directories, not files, when it pre-creates mount points (so that Docker won't attempt to and fail due to a RO volume).

I think that the issue is that once a mount point is created as a directory within a configmap, Docker will fail to bind-mount a file over the top of the directory. If that's true, I'm not sure how we can fix this without being told not to create a nested mount point.

Ugly! It's tempting to agree with @sjenning that we should just draw a line in the sand and say that we don't support nested volumes beneath API data volumes.

@Joseph-Irving until we figure out a way to fix this, as a work-around, you may need to mount the config map elsewhere and create symlinks to its data.

Edit: I wrote all of the above without actually trying anything out or looking at any code so I may be totally wrong in my interpretation. I won't have time to look at it until next week.

[msau42]

Yes, when you bind mount, the target has to be the same file type as the source

[Joseph-Irving]

Hi, I was just wondering you had made any decisions in regard to this? If you've decided to not support this I think it would make sense to have a validation check on the API that will prevent you from creating a subPath in the same path as another volume, or at least catch the error and have it return something a bit more readable to prevent confusion.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Hashfyre]

We are seeing the same issue:

Versions:

• Kops: v1.10.0
• Cloud Provider: AWS
• filebeat: docker.elastic.co/beats/filebeat:6.3.0
  • uname -a
    • Linux ip-10-110-51-197 4.4.0-1054-aws Added the example dockerfiles. #63-Ubuntu SMP Wed Mar 28 19:42:42 UTC 2018 x86_64 Linux
• kubernetes:
  server: v1.12.1
  Client: v1.12.2
• docker: 17.03.2-ce, build f5ec1e2

Error causing the crashLoop

container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/kubelet/pods/ee4f5c4b-1d41-11e9-ad17-0a422bf36948/volume-subpaths/config/filebeat/0\\\" to rootfs \\\"/var/lib/docker/overlay2/45773094b37e4019dda5f337c99d0f9c12ed6734db4ba613eb1ed6d2f413f66b/merged\\\" at \\\"/var/lib/docker/overlay2/45773094b37e4019dda5f337c99d0f9c12ed6734db4ba613eb1ed6d2f413f66b/merged/etc/filebeat.yml\\\" caused \\\"no such file or directory\\\"\""

@nikhita could this be re-opened, is there a known fix, or a duplicate issue I should track?

[nikhita]

@nikhita could this be re-opened, is there a known fix, or a duplicate issue I should track?

@Hashfyre I don't think there are any plans to address this in the near future because it would warrant an api change. Ref: #61563 (comment).