Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch from Tensorflow Serving to Triton #622

Closed
mithridatea opened this issue Feb 14, 2022 · 2 comments
Closed

Switch from Tensorflow Serving to Triton #622

mithridatea opened this issue Feb 14, 2022 · 2 comments
Assignees
@raphael0202
Labels
MLOps

Comments

@mithridatea
Copy link
Contributor

We currently use Tensorflow Serving to serve all models:

  • object detection models (nutriscore, universal-logo,...)
  • category classifier model

However, Tensorflow Serving only supports the SavedModel format, which is specific to Tensorflow. It means we cannot easily integrate models trained with other frameworks (pytorch, scikit-learn,...).

ONNX is a standard format to save model architecture and weights supported by all major frameworks, including Tensorflow, Pytorch and scikit-learn.

Proposal

Test Triton, an open-source inference server developed by Nvidia. Supported formats include TF SavedModel and ONNX. Runtime on both GPU and CPU.
If tests are successful, switch from Tensorflow Serving to Triton server.
@ram690514
Copy link

I am interested to work on this issue , if no one is currently working on it

@github-actions github-actions bot added the ⭐ top issue Top issue. label Sep 20, 2022
@github-actions github-actions bot mentioned this issue Sep 20, 2022
Open
@teolemon teolemon moved this to To discuss and validate in 🤖 Artificial Intelligence @ Open Food Facts Sep 20, 2022
@raphael0202 raphael0202 self-assigned this Oct 27, 2022
@raphael0202 raphael0202 mentioned this issue Oct 28, 2022
Closed
raphael0202 added a commit that referenced this issue Oct 28, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
633019e 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
raphael0202 added a commit that referenced this issue Oct 31, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
b39e925 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
raphael0202 added a commit that referenced this issue Nov 8, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
310f05f 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
raphael0202 added a commit that referenced this issue Nov 8, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
47f31ff 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
raphael0202 added a commit that referenced this issue Nov 8, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
4c83189 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
raphael0202 added a commit that referenced this issue Nov 8, 2022
@raphael0202
feat: switch all object detection models to Triton (#622)
3c786c4 
- switch to Triton, use gRPC protocol for faster transfer
- only request needed output fields (resolves #942)
- get rid of CategoryIndex and protobuf-generated files
@alexgarel alexgarel mentioned this issue Nov 8, 2022
Merged
@raphael0202
Copy link
Collaborator

The switch to Triton is done in preprod, closing this issue.

Repository owner moved this from To discuss and validate to Done in 🤖 Artificial Intelligence @ Open Food Facts Nov 8, 2022
@raphael0202 raphael0202 removed the ⭐ top issue Top issue. label Apr 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raphael0202 raphael0202

Labels
MLOps
Projects
🤖 Artificial Intelligence @ Open Food Facts
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@raphael0202 @ram690514 @mithridatea