Pretrained Models

[silver-fox-12]

I wonder if it aligns with the project goals to support the integration of pretrained models.

A possible solution could be to host the pretrained models on the Huggingface Model Hub. It would then be possible to build a wrapper that loads a model from the hub & directly use it for inference to lower the entry barrier for noobs like me 🙂

[wyli]

duplicate of #271. Thanks for the suggestions, I'm converting this into a discussion.

Some of the models in this project already support loading from the pretrained, please see:

• /~https://github.com/Project-MONAI/MONAI/search?q=pretrained
• /~https://github.com/Project-MONAI/tutorials/blob/master/modules/transfer_mmar.ipynb
• /~https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/mmars/model_desc.py

• MONAI/monai/apps/mmars/mmars.py

  Lines 207 to 210 in 29e9ab3

  	Examples::
  	>>> from monai.apps import load_from_mmar
  	>>> unet_model = load_from_mmar("clara_pt_prostate_mri_segmentation_1", mmar_dir=".", map_location="cpu")
  	>>> print(unet_model)

[omarespejel]

Hi Monai team (cc @wyli)!

Continuing this Discussion, I wanted to formally propose the integration with the Hugging Face Hub.

At Hugging Face we are aware of the relevance of Monai in deep learning for healthcare and would love to contribute with you to this effort.

This integration would allow you to freely download/upload models from/to the Hugging Face Hub: https://huggingface.co/.

I see you currently save your model checkpoints through the NGC Catalogue and the MMAR. Would you be interested in sharing your pre-trained models such as clara_pt_brain_mri_annotation_t1c in the Hugging Face Hub?

We can help you set up a Monai organization (examples, Facebook AI y Stanford NLP).

The Hub offers free hosting of over 25K models, and it would make your work more accessible and visible to the rest of the ML community.

Some of the benefits of sharing your models through the Hub would be:

• As @silver-fox-12 argues, presence in the HF Hub might lower the entry of barrier to MONAI as well as increase its visibility.
  • Repos provide useful metadata about their tasks, languages, metrics, etc that make them discoverable
• versioning, commit history and diffs
• multiple features from TensorBoard visualizations, PapersWithCode integration, and more

Creating the repos and adding new models should be a relatively straightforward process if you've used Git before. This is a step-by-step guide explaining the process in case you're interested. Please let us know if you would be interested and if you have any questions.

Happy to hear your thoughts,
Omar and the Hugging Face team

cc @diazandr3s @osanseviero @abidlabs

[osanseviero]

As an additional point, we have a library to programmatically access repositories (both downloading pretrained models and pushing models as well, with a lot of nice things such as filtering, caching, etc). If we want to try out this integration, I would suggest taking a first step:

• Add one or two models manually
• Use huggingface_hub library to implement downloading those models programmatically from monai. You might want to check https://huggingface.co/docs/hub/adding-a-library to read more about it, but we would be happy to work on it and open a draft PR if you're interested.

[wyli]

thanks for the great proposals, I'll bring them up at the dev meeting and get back to you.

[omarespejel]

Thank you wyli. Please let me know anything additional that you need.

[wyli]

Thank you, I've created an org https://huggingface.co/MONAI, we'll probably upload model examples as part of #3482 (ongoing discussions about monai model format) and look into the python APIs for model downloading. We'll get back to you with further questions if there's any @osanseviero @omarespejel!

[omarespejel]

That is great @wyli! Sure, feel free to tell us if you have any questions or want to review something.

As a side note, on the left side filter bar on the Hub Models webpage we have a section to filter by library (screenshot below). It would be useful to have a monai tag on the models you load. This will automatically be recognized as a library with a small PR that adds a code sample on how to load the models.

[omarespejel]

Hi @wyli! Hope you are well.

We are very excited about collaborating with you to share MONAI models with the 🤗 HF community. With the possibility of also allowing MONAI users to quickly share and load models from the HF Hub (adding push_to_hub and load/download_from_hub functionalities within MONAI). Possibly makes sense according to issue #3482.

I can help you with what you need.

If you wish I can create a Slack channel (#MONAI-hf) where we can support and help you with what you need. Would this be a bad idea?

(cc @Nic-Ma)

Kind regards!

[Nic-Ma]

Hi @omarespejel ,

Thanks for your help. We are still working on #3482 , you can see the links of PRs in the ticket description.
Once the features are ready, I think we will investigate how to upload to huggingface (the 10th task in the list).
Please feel free to create a slack channel if you want, I also suggest to keep discussing in Github, so everyone can see the progress.

Thanks in advance.

[omarespejel]

Hi @Nic-Ma, thanks for your kind answer. Makes sense to me. I agree we can continue discussing this here.

Please let me know if you need anything :)

Thanks!

[Nic-Ma]

Hi @omarespejel ,

I am trying to create an account on huggingface, but blocked by below page, the "create account" button doesn't work:

I am using Chrome browser at China, do you know any possible reason?

Thanks.

[omarespejel]

Hi @Nic-Ma! That is odd. Let me bring it to the team and circle back.

[Nic-Ma]

Hi @omarespejel ,

I figured out the root cause, when I changed to a very complicated password, it can create an account now.
I think maybe you can enhance the web page to provide some warning or tips if the password doesn't match the format.

Thanks.

[omarespejel]

Hi @Nic-Ma, It's great that you were able to access Hugging Face! I will communicate your feedback to our team. Thanks!

Please let me know if you need anything else.

Kind regards.

[Nic-Ma]

Hi @omarespejel ,

I uploaded an example bundle model to: https://huggingface.co/MONAI/example_spleen_segmentation.
And I added the tag monai for the model.
During uploading, I got 2 questions:

1. How can we upload multiple files together? seems I can't drag several files then commit.
2. Does it support folders? Seems we can't upload structured folders, like: models, configs, docs, etc.

Thanks.

[Nic-Ma]

Hi @omarespejel ,

Thanks for your continuous support, I shared the draft example in our internal meeting.
I have 2 questions before we dive deeper into the advanced features:

1. How can we review & approve l if someone submitted a new model to the MONAI organization in huggingface?
   I think it should be something like a pull request in Github?
2. How to set up and execute CI / CD tests everytime submitting a new model to huggingface?
   Especially a new model under review.

Thanks in advance.

[omarespejel]

Hi @Nic-Ma, sorry for the late reply. You are welcome! We are very happy with the possibility of sharing MONAI models with the HF community 🤗.

1. Only people that are members of the MONAI organization in HF can submit models to the MONAI organization. Currently, we do not have functionality such as PRs. However:
   1.1. Users can create a MONAI model in their own account and then contact the MONAI organization to transfer the model from their individual account to the MONAI organization account.
   1.2. Not all models need to be in the MONAI organization. Third users can create models based on a MONAI pre-trained model and use the monai tag. For example, the speechbrain and spacy libraries have their own organizations but also HF users create their own models with their respective tags (here for speechbrain and here for spacy).

2. Unfortunately, CI / CD tests are not available in HF models.

I hope this has been helpful! And please let me know any other questions you have. We would love to push this integration.

Kind regards.

[Nic-Ma]

CC @ericspod @wyli

Thanks.

[ericspod]

This is looking like a good partnership going forward! It would help if we could upload a zip file containing the files in their correct folders to be unzipped into the files stored in HF. We can check the repos out and rejig locations but this seems like it would be easier. What we can do instead on our end is provide scripts to automate the process of checkout-upload-commit to streamline things, and we should look into integrating our own CI/CD process into checking models are correct. Somehow we can use Github Actions to do what we need I think. Other users being able to tag monai on their own models is a good solution that will allow us to be recognised with models we don't necessarily want to host in our own organisation but still pulls them into our community.

[osanseviero]

Sounds amazing 🚀

I'm not sure if you've checked the huggingface_hub library, but it should also allow you to programmatically upload models to the Hub without having to handle all the git stuff yourselves, it handles most of it 😄 . See https://huggingface.co/docs/hub/adding-a-model#using-the-huggingface_hub-client-library as a minimal guide.

[Nic-Ma]

Mark: a related task tracking ticket related to this topic: #4042

Thanks.

[yiheng-wang-nv]

Hi @omarespejel , here I wish to ask a question about the download part of hugging face.
I took some time to look at the source code of transformers, and found that we can use the following way to download a specific file:

from transformers import cached_path
from transformers import configuration_utils

config_file = \
configuration_utils.hf_bucket_url(model_id="MONAI/example_spleen_segmentation", filename="model.pt")
resolved_config_file = cached_path(config_file, cache_dir="weights")

However, I'm not sure if hugging face support to download a whole folder directly? The only way I found is to use: git clone, is there any other methods, such as using python API? Thanks!

[osanseviero]

Hey there!

Instead of using transformers, you can use huggingface_hub which allows you to easily download and cache model and other files from the Hub. You can refer to https://huggingface.co/docs/hub/how-to-downstream for help, hf_hub_download can help you download a single file and snapshot_download allows you to download the whole repository.

[omarespejel]

Hi @Nic-Ma @wyli @ericspod.

Regarding a previous question from @Nic-Ma, we have new collaboration features in the Hub: pull requests and discussions (link to announcement).

This would allow MONAI users to submit, edit, or discuss models within the MONAI organization.

This is in addition to what @ericspod mentioned: other users can tag monai on their own models so you can recognize models that you don't want to host in your own organization, but still engage with your community.

Also, if you want, we can do a PR to allow image segmentation inference directly from the browser like for example with DETR.

Please let me know if we can help further.

[Nic-Ma]

Hi @omarespejel ,

Cool! Glad to see this discussion and PR support in huggingface.
To enable the model contribution pipeline, we are building a new repo in Project-MONAI to do the review and CICD of model PRs.
We may try to investigate more huggingface features when the repo is ready for public use.

Thanks.

[omarespejel]

Hi @Nic-Ma! Thank you for your kind message!

That's great! Sure, let us know if you need help when investigating the HF features. We are happy to help further push the MONAI models usage through the HF Hub. DL for healthcare is key for our mission of democratizing ML.

[ericspod]

Definitely a great addition, I saw the announcement previously and thought this was a great direction for the sort of community organisation we'd like.

[omarespejel]

Great to hear that @ericspod! That is our goal with that release, to empower the community.