This repository contains a simple machine learning workflow consisting of data ingestion and preparation, model training, serving and monitoring. It represents how computer-aided diagnosis can be used for the prediction of pneumonia from a collection (a volume) of CT images.
It is based on work by Hasib Zunair.
- Openshift
- OpenDataHub
- Jupyter Notebooks with iPyWidgets
- Numpy
- Tensorflow
- Python requests library
- Seldon Core
- Prometheus
- Grafana
├── 01-inference-3d-image-classification-cli.py Python script forinferencing
├── 01-inference-3d-image-classification.ipynb Visualization and inferencing
├── 02-training-3d-image-classification.ipynb Notebook script for training
├── 02-training-3d-image-classification.py Python script for training
├── 3d_image_classification.h5 Trained model artifact
├── Dockerfile For s2i builds
├── MyModel.py Seldon Model Server Code
├── ct-data.zip Validation data for inferencing
├── requirements-notebook.txt
├── requirements.txt
└── resources Kubernetes Objects
├── 06-seldon-mymodel-servicemonitor.yaml
├── 07-mymodel-seldon-deploy-from-quay.yaml
└── grafana-dashboards
├── NVIDIA-DCGM-dashboard.json GPU Metrics
└── seldon-dashboard.json Model Server Metrics
Train a 17-layer, Convolutional Neural Network to predict the presence of COVID-19 related pneumonia from 3D CT imagery.
Build the training Python stack.
pip install pip tensorflow nibabel matplotlib -Uq
- (200) COVID-19 related 3D CT image studies
- Each study contains 36-54 slices of 512x512 pixels (voxels) each.
- Total size is ~2GB (compressed)
- ~20 minutes to preprocess and train on an NVIDIA Tesla T4 GPU
- ML framework: Keras/Tensorflow
Data Source: Chest CT Scans with COVID-19 Related Findings.
- Change to the
resourcesdirectory.
cd 3d-image-classification/resources
- Create a project called
ml-mon
oc new-project ml-mon
- Using the Openshift console UI, install an instance of the following community operators from OperatorHub into the
ml-monnamespace.
- OpenDataHub
- JupyterHub, S3, ODH Dashboard
- Prometheus
- Grafana
Seldon
- Install the Seldon Core operator into all namespaces in the cluster (default).
- Create an instance of Prometheus and Grafana in the
ml-monnamespace.
Expected Output
oc get pods -n ml-mon -w
NAME READY STATUS RESTARTS AGE
$ oc get pods -n ml-mon
NAME READY STATUS RESTARTS AGE
grafana-deployment-8fbf7c944-7895m 1/1 Running 0 5h35m
grafana-operator-controller-manager-6ff698d9fc-xvk28 2/2 Running 0 5h35m
prometheus-example-0 2/2 Running 0 5h35m
prometheus-operator-7b9ccd45c6-7v8td 1/1 Running 0 5h35m
Create routes for Prometheus and Grafana.
oc expose svc prometheus-operated
oc expose svc grafana-service
Obtain the Grafana admin credentials to login to the Grafana console.
oc get secrets grafana-admin-credentials -o=jsonpath='{@.data.GF_SECURITY_ADMIN_USER}' | base64 --decode
admin
oc get secrets grafana-admin-credentials -o=jsonpath='{@.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 --decode
ABcdRqpfdsEfpg==
- Create a Prometheus Service Monitor
oc create -f 06-seldon-mymodel-servicemonitor.yaml
servicemonitor.monitoring.coreos.com/mymodel-mygroup created
-
Login to the Grafana console. The username and password can be obtained from the
grafana-admin-credentialssecret. -
Within Grafana, configure a Prometheus data source called
prometheuswith a URL ofprometheus-operated.ml-mon:9090 -
Import the Seldon dashboard from the
resources/seldon-dashboard.jsonfile. -
Deploy the Seldon model server and wait for the classifier pod to become ready. Two services should be created by the Seldon deployer.
oc create -f 07-mymodel-seldon-deploy-from-quay.yaml
seldondeployment.machinelearning.seldon.io/mymodel created
oc get pods
NAME READY STATUS RESTARTS AGE
mymodel-mygroup-0-classifier-57647887d9-98qqb 2/2 Running 0 118s
oc get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mymodel-mygroup ClusterIP 10.217.5.143 <none> 8000/TCP,5001/TCP 20s
mymodel-mygroup-classifier ClusterIP 10.217.4.127 <none> 9000/TCP 2m4s
- Create a route for the Seldon model server.
oc expose svc mymodel-mygroup
Curl the prometheus endpoint and confirm it is able to scrape metrics from the classifier pod.
curl -X GET $(oc get route mymodel-mygroup -o jsonpath='{.spec.host}')/prometheus
...
promhttp_metric_handler_requests_total{code="200"} 5
Jupyter Notebook dependencies
pip install tensorflow jupyterlab ipywidgets scipy
- Login to OpenDataHub
- Start the JupyterHub server and choose the
Standard Data Sciencenotebook image. - Clone this github repo
- Run the
01-inference-3d-image-classificationnotebook. - Find the notebook cell with
predictfunction and modify theurlvariable to point to the route that was created.echo $(oc get route mymodel-mygroup -o jsonpath='{.spec.host}')/api/v1.0/predictions
- Run the notebook and select a study to make a few predictions to trigger Seldon activity.
Within 30 seconds or so there should be activity on the Seldon Grafana Dashboard.
Optionally, configure Grafana to watch Openshift's built-in Prometheus Data Source so a GPU dashboard can be created. This data source will scrape metrics from the NVIDA DCGM exporter.
Grant the Grafana service account name the cluster-reader role so it can use
Openshift's Prometheus in the openshift-monitoring namespace.
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-service-account -n ml-mon
Get the Prometheus token.
oc serviceaccounts get-token prometheus-k8s -n openshift-monitoring
Add this token to the example Grafana data source yaml.
httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
Create the data source object.
oc apply -f 03-prometheus-grafanadatasource.yaml
Import the Seldon and GPU dashboards from the included json files.
Open The Prometheus and Grafana Dashboards to visualize the API activity.
curl -X GET $(oc get route mymodel-mygroup -o jsonpath='{.spec.host}')/prometheus
seldon_api_executor_server_requests_seconds_sum{code="200",deployment_name="mymodel",method="post",predictor_name="mygroup",predictor_version="",service="predictions"} 4.714845908
seldon_api_executor_server_requests_seconds_count{code="200",deployment_name="mymodel",method="post",predictor_name="mygroup",predictor_version="",service="predictions"} 5
$ oc create -f resources/07-mymodel-seldon-deploy-from-quay.yaml
Error from server (InternalError): error when creating "resources/07-mymodel-seldon-deploy-from-quay.yaml": Internal error occurred: failed calling webhook "v1.vseldondeployment.kb.io": Post "https://seldon-webhook-service.odh.svc:443/validate-machinelearning-seldon-io-v1-seldondeployment?timeout=30s": service "seldon-webhook-service" not found
This can happen after ODH has been re-installed into a different project. To fix it delete the old webhook.
oc get MutatingWebhookConfiguration,ValidatingWebhookConfiguration -A
oc delete validatingwebhookconfiguration.admissionregistration.k8s.io/seldon-validating-webhook-configuration-odh
cd 3d-image-classification
oc new-build --strategy docker --docker-image registry.redhat.io/ubi8/python-36 --name mymodel -l app=mymodel --binary
oc start-build mymodel --from-dir=. --follow
oc get is
NAME IMAGE REPOSITORY TAGS UPDATED
mymodel image-registry.openshift-image-registry.svc:5000/bk-models/mymodel latest 7 seconds ago
Edit mymodel-seldon-deploy.yaml to confirm that the image location matches what the image stream reports. Then deploy the model server and wait for the pod to become ready.
oc apply -f resources/mymodel-seldon-deploy.yaml
oc get pods
NAME READY STATUS RESTARTS AGE
mymodel-mygroup-0-classifier-7c6b44569c-qmzk6 2/2 Running 0 61s
Expose the service
oc expose svc <svc-name>
To trigger a redeploy after a new build. This does not always work so the pod may have to be deleted.
oc patch deployment <deployment-name> -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": { \"redeploy\": \"$(date +%s)\"}}}}}"