Estargz base images don't seem to get cached if using stargz snapshotter

[danlenar]

If buildkit is configured to use the stargz snapshotter and if your Dockerfile contains a base image that was compressed with estargz, the follower layers never seem to get cached. It seems that lazy pulling images never make into the buildkit cache.

I am able to reproduce this issue with this Dockerfile

FROM ghcr.io/stargz-containers/golang:1.18-esgz

RUN echo "True"

No matter how many times I build this, the layer 'RUN echo "True"' will never get cached.

docker build --platform=linux/amd64 --progress=plain .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 102B done
#2 DONE 0.0s

#3 [internal] load metadata for ghcr.io/stargz-containers/golang:1.18-esgz
#3 DONE 0.2s

#4 [1/2] FROM ghcr.io/stargz-containers/golang:1.18-esgz@sha256:a0aba3286b63a5c65754e26452cb8508a010e796786fbbab81fe94d9a579e471
#4 resolve ghcr.io/stargz-containers/golang:1.18-esgz@sha256:a0aba3286b63a5c65754e26452cb8508a010e796786fbbab81fe94d9a579e471 done
#4 DONE 0.0s

#5 [2/2] RUN echo "True"
#5 0.171 True
#5 DONE 0.2s

If I switch to using overlayfs snapshotter, then I do see expected behavior after the second docker build

docker build --platform=linux/amd64 --progress=plain .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 102B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for ghcr.io/stargz-containers/golang:1.18-esgz
#3 DONE 0.3s

#4 [1/2] FROM ghcr.io/stargz-containers/golang:1.18-esgz@sha256:a0aba3286b63a5c65754e26452cb8508a010e796786fbbab81fe94d9a579e471
#4 DONE 0.0s

#5 [2/2] RUN echo "True"
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:de12d0d31b46be378e61cca83bcd76887552d0e27bfc54ed486aeb5ce1b15297 done
#6 DONE 0.0s

I am able to replicate this on Docker Desktop on Mac by having the "Use containerd for pulling and storing images" option enabled. I can also replicate this using a remote buildkit (v0.11.6).

My buildkitd.toml config

    debug = true

    [grpc]
     address = [ "unix:///run/buildkit/buildkitd.sock", "tcp://0.0.0.0:1234" ]

    [worker.oci]
      enabled = true
      gc = true
      gckeepstorage = 1717986918400
      snapshotter = "stargz"

[ktock]

@danlenar This was fixed by #3109 . Could you try BuildKit newer than that commit or the latest one on the master branch?