[feature] Update MLMD version and add process to keep this up to date.

[davidspek]

Currently pipelines uses gcr.io/tfx-oss-public/ml_metadata_store_server version 0.25.1 but 0.29.0 is the latest version. The image tag in metadata-grpc-deployment.yaml needs updating.
To automate this process, all image tags should be listed in the images: section of the kustiomization.yaml files (here for this specific case). Once that is done, Renovate can be setup to create PRs updating the image tags of the manifests.

@Bobgy Would you like me to go through the manifests and set the images in the kustomization.yaml files and add the necessary Renovate config for kustomize?

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[Bobgy]

Note, updating MLMD is fairly complex, we need to update all the 7 locations listed in

pipelines/manifests/kustomize/base/metadata/base/metadata-grpc-deployment.yaml

Lines 19 to 24 in 1128631

	# ! Sync to the same MLMD version:
	# * backend/metadata_writer/requirements.in and requirements.txt
	# * @kubeflow/frontend/src/mlmd/generated
	# * .cloudbuild.yaml and .release.cloudbuild.yaml
	# * manifests/kustomize/base/metadata/base/metadata-grpc-deployment.yaml
	# * test/tag_for_hosted.sh

Some of them are not well documented, so I'd suggest delaying the upgrade to next release -- unless you have absolute reasons.

[davidspek]

@Bobgy I have no particular reason for updating it, just something I noticed. Once things settle down a bit more I think it would be interesting to setup Renovate with the regex engine so that it can handle updating MLMD everywhere in the repo in a single PR so it can be tested properly. How does that sound?

[Bobgy]

Part of the upgrade requires regenerating a proto client using later proto definitions, so it's unlikely we can configure renovate to do this.

But it's very helpful to automate the upgrade, we may add this script to release process

[davidspek]

You can have Renovate execute a script as well, so it might still be possible to fully automate.

[davidspek]

Happy to see that theMLMD version has been upgraded in the latest release. @Bobgy is having this managed by Renovate something you would want to implement? I've been doing quite a bit of stuff with Renovate lately to automate keeping all the manifests for Kubeflow up to date so I might be able to help here.

[Bobgy]

@davidspek did you figure out the scriptable update? MLMD would need this, because one step is pulling in new proto definitions and regenerate proto clients.

[davidspek]

Do you maybe have a link to the script that needs to be run and the locations where MLMD needs to be updated? It would save me some time looking for it. If not, it isn't a problem either. Then I can test on my fork to try and get this setup.
Renovate has support for running scripts if it is self-hosted, but I'm not sure if that is an option or a path you want to go down. However, I think it might be possible to use another setup to run that script or pull those definitions into the PR.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

/cc @zijianjoy
Your new script addresses first part of this issue

[zijianjoy]

Thank you David and Yuan! This is the link to the MLMD upgrade script guide: /~https://github.com/kubeflow/pipelines/tree/master/third_party/ml-metadata#upgrade-mlmd-versions

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thesuperzapper]

@zijianjoy @Bobgy @chensun, I see that we are now very behind again on the ml-metadata version, we are still running 1.5.0 from 2021, but 1.13.1 exists.

Any chance someone can manually do another update?

[zijianjoy]

We are currently waiting for ML Metadata 1.14 to be released. Once it is available, we will upgrade to this version.

[rimolive]

Closing this issue as ML Metadata 1.14 is already released and manifests upgraded.

/close

[google-oss-prow]

@rimolive: Closing this issue.

In response to this:

Closing this issue as ML Metadata 1.14 is already released and manifests upgraded.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.