Age Out Old Registry Content

[BenTheElder]

We migrated forward all existing images since the beginning of hosting Kubernetes images, from gcr.io/google-containers to k8s.gcr.io to registry.k8s.io

We should consider establishing an intended retention policy for images hosted by registry.k8s.io, and communicating that early.

Not retaining all images indefinitely could help the container-image-promoter avoid dealing with an indefinitely growing set of images to synchronize and sign and may also have minor hosting cost benefits.

Even if we set a very lengthy period, we should probably consider not hosting content indefinitely.

[jeefy]

Jeefy lizard brain policy idea:

Non-Prod images: Age out after 9 months
Prod images: Age out after release-EOL + 1y (so if 1.27 aged out Jan 2024, it would get culled Jan 2025)

[mrbobbytables]

I am broadly in support of this. I don't think its a reasonable ask for the project to host all releases for all time.
Do we have any data on which versions are being pulled?
I know there are 3rd party reports available (e.g. datadog report) that show 1.21 is still the most common version deployed right now, I would want to make sure we take that into account.

Right now I'm leaning towards EOL-9 (3 years), but would want some data before making any decision.

EDIT: Even without data I think we should remove the google-containers images...oof

[ameukam]

/sig testing
/kind cleanup

[BenTheElder]

Some additional points:

• We have a lot of images that are not related to Kubernetes releases, but may have their own release timelines. I think we should maybe just set an N year policy, where N is relatively generous but still gives us room to stop perma-hosting.

• The mechanism to remove images needs some thinking ...

  • The source of truth on what images are available is in the /~https://github.com/kubernetes/k8s.io repo, and is /~https://github.com/kubernetes-sigs/promo-tools copying to the backing registries (so to be clear no changes will happen in this repo, it's a multi-repo problem and I figured visibility might be best here, but we need to forward this to more people).
  • We might have to cull them from the image promoter manifests somehow, and then start to permit auto-pruning things that are in production storage but not in the manifests. Automating the policy to drop things from the source manifests sounds tricky, but I'm not sure there's a more reasonable approach. The implementation details need input from promo-tools maintainers and will probably at least somewhat drive the viable policy. cc @puerco @kubernetes/release-engineering (EDIT: see RFC: Feasibility of aging out old content kubernetes-sigs/promo-tools#719)

• IMHO Despite needing to consider the mechanics, we should decide if we're doing this and on a reasonable policy and starting communication ahead of actually implementing, it may take time to staff these changes but communicating sooner alongside the new registry would be beneficial to users.

EDIT: Even without data I think we should remove the google-containers images...oof

Yeah. That also just hasn't happened due to lack of policy. We did the flip to k8s.gcr.io July 24th 2020. kubernetes/release#270 (comment)

/sig testing

While I'm sure sig-testing is happy to support this effort, I'd suggest that the policy is a combination of k8s-infra (what is k8s-infra willing to fund resources for generally) and release (particularly around the promo tools support for this and kubernetes release timeframe).

/sig k8s-infra
/sig release
/remove-sig testing

[mrbobbytables]

/committee steering
as this has budget / cncf related items

[sftim]

Technical aside: when we serve layer redirects, we then get an option to serve a Warning header alongside the redirect.

[BenTheElder]

One additional complication for people to consider: We more commonly host our own base images at this point, building old commits from source will become more challenging (but not necessarily impossible*) if we age out those images.

* non-reproducible builds may be an issue, e.g. the "debian-base" image.