Impossible to delete an image #15006

slemeur · 2022-07-21T08:58:52Z

/kind bug

Description

I have an image that I can't delete because Podman says it's currently in use by a container which does not exist.

Steps to reproduce the issue:

Unfortunately, I'm not sure how I ended up in this situation.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version                                                                                                                 1 ✘  16m 26s 
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Tue Jun 14 22:12:46 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.16.14
Built:        Thu Mar  3 15:56:56 2022
OS/Arch:      linux/amd64

Output of podman images:

REPOSITORY                                   TAG                  IMAGE ID      CREATED        SIZE
docker.io/library/my-custom-image            latest               5be24218919b  7 minutes ago  578 MB
quay.io/podman/hello                         latest               9b0445929568  7 days ago     81.8 kB
<none>                                       <none>               1b6b53283e24  2 months ago   1.89 GB
docker.io/library/httpd                      latest               6b8e87fff107  4 months ago   148 MB
docker.io/bitnami/mariadb                    10.5.9-debian-10-r0  2df089982a8d  17 months ago  336 MB
quay.io/quarkus/centos-quarkus-maven         20.0.0-java11        b3e39ba63244  23 months ago  1.89 GB
registry.access.redhat.com/ubi8/ubi-minimal  8.1-409              91d23a64fdf2  2 years ago    109 MB

Output of podman rmi:

podman rmi 1b6b53283e24                                              ✔ 
Error: Image used by 8ed9a6770b15351d30638e95729fe076d50ac89ac7ddfffa3efe3e855ac06506: image is in use by a container

Output of podman image inspect:

podman image inspect 1b6b53283e24                                                                        ✔ 
[
     {
          "Id": "1b6b53283e24f83803da6d5a5418b528670ab7f159bf69f0f25ecfcb23f11cb4",
          "Digest": "sha256:02fc108b1fe31621897182b5832fb3bb33e94e4c30fdf8aa3deff9361257a3f0",
          "RepoTags": [],
          "RepoDigests": [],
          "Parent": "4ef0b4011cf5ed78d20756f707a073cce66f75bf1025e011faf7706e01a4bd59",
          "Comment": "",
          "Created": "2022-05-03T14:58:14.919087242Z",
          "Config": {
               "User": "quarkus",
               "ExposedPorts": {
                    "8080/tcp": {}
               },
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/graalvm/bin",
                    "LANG=en_US.UTF-8",
                    "LANGUAGE=en_US:en",
                    "LC_ALL=en_US.UTF-8",
                    "QUARKUS_HOME=/home/quarkus",
                    "MAVEN_HOME=/usr/share/maven",
                    "MAVEN_OPTS=-XX:+TieredCompilation -XX:TieredStopAtLevel=1",
                    "MAVEN_VERSION=3.6.3",
                    "GRADLE_HOME=/usr/share/gradle",
                    "GRADLE_OPTS=-Dorg.gradle.daemon=false",
                    "GRADLE_VERSION=6.3",
                    "FILENAME=graalvm-ce-linux-amd64-20.0.0-java11.tar.gz",
                    "GRAALVM_HOME=/opt/graalvm",
                    "GRAALVM_VERSION=java11-20.0.0",
                    "JAVA_HOME=/opt/graalvm"
               ],
               "Entrypoint": [
                    "/usr/local/bin/entrypoint-run.sh"
               ],
               "Cmd": [
                    "mvn",
                    "-v"
               ],
               "WorkingDir": "/project",
               "Labels": {
                    "graalvm-archive-filename": "graalvm-ce-linux-amd64-20.0.0-java11.tar.gz",
                    "graalvm-archive-url": "/~https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-20.0.0/graalvm-ce-java11-linux-amd64-20.0.0.tar.gz",
                    "graalvm-version": "java11-20.0.0",
                    "io.buildah.version": "1.24.1",
                    "io.cekit.version": "3.7.0",
                    "io.k8s.description": "Quarkus.io builder image for building Quarkus applications.",
                    "io.k8s.display-name": "Centos + GraalVM + Maven Image",
                    "io.openshift.expose-services": "8080:http",
                    "io.openshift.tags": "builder,java,quarkus",
                    "maintainer": "Quarkus Team <quarkus-dev@googlegroups.com>",
                    "org.label-schema.build-date": "20200809",
                    "org.label-schema.license": "GPLv2",
                    "org.label-schema.name": "CentOS Base Image",
                    "org.label-schema.schema-version": "1.0",
                    "org.label-schema.vendor": "CentOS",
                    "org.opencontainers.image.created": "2020-08-09 00:00:00+01:00",
                    "org.opencontainers.image.licenses": "GPL-2.0-only",
                    "org.opencontainers.image.title": "CentOS Base Image",
                    "org.opencontainers.image.vendor": "CentOS"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "amd64",
          "Os": "linux",
          "Size": 1889395209,
          "VirtualSize": 1889395209,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/17f08ab058211a61949b72e535871db9e9d736d715f57eb03ff638fc6b4217ba/diff:/var/home/core/.local/share/containers/storage/overlay/9e4f8b7d485422648a3b783ff7af81dda4cc72fc349317ecac2d6e445458f876/diff:/var/home/core/.local/share/containers/storage/overlay/2ed0b88d698a5b80d1e6d2e01c54626d9325ceab2a87f876050b561d03c25eaa/diff:/var/home/core/.local/share/containers/storage/overlay/613be09ab3c0860a5216936f412f09927947012f86bfa89b263dfa087a725f81/diff",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/9d8cd1578897d624d16714529fe8a874525aa5a6b6cbbeea8836dadcdc0c9c86/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/9d8cd1578897d624d16714529fe8a874525aa5a6b6cbbeea8836dadcdc0c9c86/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:613be09ab3c0860a5216936f412f09927947012f86bfa89b263dfa087a725f81",
                    "sha256:09bc1614a6b1e7dce4dbf68ebf3080d744b1a5a31a1cd75b867d8a82b263839b",
                    "sha256:dcc99d43921d1a06481595c055d6c1548af2018f386a8f0784581b5976bbc91d",
                    "sha256:9093d3ccc79eb49d95b7fffcc660e7884d17e4356c02d089a5814e624d9e4c4f",
                    "sha256:cbf51ce405b71d4341a990bf1a72c7189fb60d0357e3eab923728dd26593e01e"
               ]
          },
          "Labels": {
               "graalvm-archive-filename": "graalvm-ce-linux-amd64-20.0.0-java11.tar.gz",
               "graalvm-archive-url": "/~https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-20.0.0/graalvm-ce-java11-linux-amd64-20.0.0.tar.gz",
               "graalvm-version": "java11-20.0.0",
               "io.buildah.version": "1.24.1",
               "io.cekit.version": "3.7.0",
               "io.k8s.description": "Quarkus.io builder image for building Quarkus applications.",
               "io.k8s.display-name": "Centos + GraalVM + Maven Image",
               "io.openshift.expose-services": "8080:http",
               "io.openshift.tags": "builder,java,quarkus",
               "maintainer": "Quarkus Team <quarkus-dev@googlegroups.com>",
               "org.label-schema.build-date": "20200809",
               "org.label-schema.license": "GPLv2",
               "org.label-schema.name": "CentOS Base Image",
               "org.label-schema.schema-version": "1.0",
               "org.label-schema.vendor": "CentOS",
               "org.opencontainers.image.created": "2020-08-09 00:00:00+01:00",
               "org.opencontainers.image.licenses": "GPL-2.0-only",
               "org.opencontainers.image.title": "CentOS Base Image",
               "org.opencontainers.image.vendor": "CentOS"
          },
          "Annotations": {},
          "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",
          "User": "quarkus",
          "History": [
               {
                    "created": "2020-08-10T18:20:08.948813347Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:61908381d3142ffba798ae9a904476d19b197ab79d0338f14bec0f76649df8d4 in / "
               },
               {
                    "created": "2020-08-10T18:20:09.298928893Z",
                    "created_by": "/bin/sh -c #(nop)  LABEL org.label-schema.schema-version=1.0 org.label-schema.name=CentOS Base Image org.label-schema.vendor=CentOS org.label-schema.license=GPLv2 org.label-schema.build-date=20200809 org.opencontainers.image.title=CentOS Base Image org.opencontainers.image.vendor=CentOS org.opencontainers.image.licenses=GPL-2.0-only org.opencontainers.image.created=2020-08-09 00:00:00+01:00",
                    "empty_layer": true
               },
               {
                    "created": "2020-08-10T18:20:09.474278304Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
                    "empty_layer": true
               },
               {
                    "created": "2020-08-23T01:31:38.706392Z"
               },
               {
                    "created": "2022-05-03T14:58:12.830020363Z",
                    "created_by": "/bin/sh -c #(nop) COPY dir:d3f231e9fb7da6c1dbbffe52d4d8c24610148c1bc81c4e9dec552f712bb84314 in /usr/src/app/src ",
                    "comment": "FROM quay.io/quarkus/centos-quarkus-maven:20.0.0-java11"
               },
               {
                    "created": "2022-05-03T14:58:13.539798645Z",
                    "created_by": "/bin/sh -c #(nop) COPY file:fe7e9e3df41c0be0c6a866c0c8e1ae66fae41795e4d806cd4e5c080e9794c833 in /usr/src/app "
               },
               {
                    "created": "2022-05-03T14:58:13.748106989Z",
                    "created_by": "/bin/sh -c #(nop) USER root",
                    "empty_layer": true
               },
               {
                    "created": "2022-05-03T14:58:14.69111845Z",
                    "created_by": "/bin/sh -c chown -R quarkus /usr/src/app"
               },
               {
                    "created": "2022-05-03T14:58:14.919696287Z",
                    "created_by": "/bin/sh -c #(nop) USER quarkus",
                    "empty_layer": true
               }
          ],
          "NamesHistory": [
               "docker.io/library/61f2f5fcc5ddcc045c8aeae8fd2929a500e40612eb6cebfae6f2c60c3a372fb5-tmp:latest"
          ]
     }
]

Output of podman ps -a:

empty list of containers

The text was updated successfully, but these errors were encountered:

Luap99 · 2022-07-21T09:06:25Z

Check with podman ps --external to see all containers including build container.
I think you can always remove the image with podman rmi --force

vrothberg · 2022-07-21T10:40:39Z

Yes, --force will remove containers (and pods) using the image.

vrothberg · 2022-08-04T12:25:24Z

@slemeur did using --force resolve the issue?

It can happen that a so-called "build container" from podman build is still hanging around and using the image. This may happen when, for instance, a build gets killed.

slemeur · 2022-08-04T12:27:08Z

Yes, I think it's exactly what happened indeed. with the --force option, I have been able to delete it. Thanks for helping out!

…

On Thu, Aug 4, 2022 at 2:25 PM Valentin Rothberg ***@***.***> wrote: @slemeur </~https://github.com/slemeur> did using --force resolve the issue? It can happen that a so-called "build container" from podman build is still hanging around and using the image. This may happen when, for instance, a build gets killed. — Reply to this email directly, view it on GitHub <#15006 (comment)>, or unsubscribe </~https://github.com/notifications/unsubscribe-auth/AAMPTIKYU5NRVOBNS7ZBWJDVXOZD5ANCNFSM54G2LRUA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

vrothberg · 2022-08-04T12:29:57Z

Glad to be of help. I was about to close the issue but I think we can improve the error message to guide users to force-removing images in that case.

vrothberg · 2022-08-04T13:00:31Z

Removing the macos label as it applies to all clients and OSes.

Trying to remove an image that is in use by containers is a confusing experience for users who may not be aware of "external" containers which are not displayed in `podman ps` by default (see containers/podman/issues/15006). Add some context to the error from containers/storage to guide the user into listing external containers and force-removing the image. Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

Improve the error message when attempting to remove an image that is in use by an external/build container. Prior, the error only indicated that the image was in use but did not aid in resolving the issue. Fixes: containers#15006 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 21, 2022

github-actions bot added the macos MacOS (OSX) related label Jul 21, 2022

benoitf added the podman-desktop label Jul 28, 2022

vrothberg self-assigned this Aug 4, 2022

vrothberg removed the macos MacOS (OSX) related label Aug 4, 2022

vrothberg mentioned this issue Aug 4, 2022

libimage: image removal: add hint to external containers containers/common#1111

Merged

vrothberg mentioned this issue Aug 4, 2022

podman rmi: improve error message for build containers #15197

Merged

openshift-ci bot closed this as completed in #15197 Aug 4, 2022

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Impossible to delete an image #15006

Impossible to delete an image #15006

slemeur commented Jul 21, 2022 •

edited

Loading

Luap99 commented Jul 21, 2022

vrothberg commented Jul 21, 2022

vrothberg commented Aug 4, 2022

slemeur commented Aug 4, 2022 via email

vrothberg commented Aug 4, 2022

vrothberg commented Aug 4, 2022

Impossible to delete an image #15006

Impossible to delete an image #15006

Comments

slemeur commented Jul 21, 2022 • edited Loading

Luap99 commented Jul 21, 2022

vrothberg commented Jul 21, 2022

vrothberg commented Aug 4, 2022

slemeur commented Aug 4, 2022 via email

vrothberg commented Aug 4, 2022

vrothberg commented Aug 4, 2022

slemeur commented Jul 21, 2022 •

edited

Loading