Edge only inference with cloud training #101

f-wright · 2024-10-07T23:17:29Z

[COM-1567] This PR adds an edge_only_inference mode which allows the model to escalate to the cloud for future training, while still always returning the edge inference answer for fast results.

Tested with 3 binary detectors with

edge_only enabled -- nothing escalated to cloud
edge_only_inference enabled -- low confidence IQs escalated to cloud, edge answer returned
Neither edge_only nor edge_only_inference enabled -- low confidence edge IQs escalated to cloud and cloud answer returned.
edge_only and edge_only_inference enabled -- pods do not launch, logs show the config validation error

Currently not adding many unit tests as they aren't set up to test post_image_query, and we want to get this change out quickly (discussed with Tyler).

This is a first step -- in the future we may want to add:

A task queueing system, instead of using FastAPI background tasks
Rate limiting for IQs sent to the cloud (either add an element of randomness for whether a query is sent to the cloud, or limit escalation to a certain number of queries over a period of time)

…dge-endpoint into edge-with-training

…edge-endpoint into edge-with-training

…dge-endpoint into edge-with-training

CoreyEWood

This looks great overall! Awesome functionality to have. I just left a few comments about making sure things aren't confusing if external users are going to be using the edge-only modes.

As an additional thought, I wonder if we should change the default refresh_rate so that it looks for new model binaries for frequently - the current default of 120 seconds is pretty slow, and users might want to see improvements more quickly if they're sending a lot of queries.

CoreyEWood · 2024-10-08T20:09:42Z

README.md

@@ -50,23 +50,32 @@ print(f"The answer is {image_query.result}")
 See the [SDK's getting started guide](https://code.groundlight.ai/python-sdk/docs/getting-started) for more info.

 ### Experimental: getting only edge model answers
-If you only want to receive answers from the edge model for a detector, you can enable edge-only mode for it. To do this, edit the detector's configuration in the [edge config file](./configs/edge-config.yaml) like so:
+If you only want to receive answers from the edge model for a detector, you can enable edge-only mode for it. This will prevent the edge endpoint from sending image queries to the cloud API. If you want fast edge answers regardless of confidence but still want the edge model to improve, you can enable edge-only inference for that detector. This mode will always return the edge model's answer, but it will also submit low confidence image queries to the cloud API for training.


I wonder if there's a different naming we could have for these different modes - I worry that the difference between "edge-only mode" and "edge-only inference" would be unclear to anyone external. Depending on how experimental/temporary these are, this might not matter, but if we expect external customers to use these at some point I think having more clear names might be helpful.

I agree that it's confusing. Maybe I could change them to edge_only_no_training and edge_only_cloud_training? That gets a little wordy, but might be more clear. Definitely open to naming suggestions, wasn't sure what to call it.

CoreyEWood · 2024-10-08T20:10:22Z

README.md

  - detector_id: 'det_abc'
    motion_detection_template: "default"
    local_inference_template: "default"
 ```
-In this example, `det_xyz` will have edge-only mode enabled because `edge_only` is set to `true`. If `edge_only` is not specified, it defaults to false, so `det_abc` will have edge-only mode disabled.
+In this example, `det_xyz` will have edge-only mode enabled because `edge_only` is set to `true`. `det_ijk` will have edge-only inference enabled because `edge_only_inference` is set to `true`. If `edge_only` or `edge_only_inference` are not specified, they defaults to false, so `det_abc` will have edge-only mode disabled. Only one of `edge_only` or `edge_only_inference` can be set to `true` for a detector.


Just a tiny typo but this says "they defaults to false".

Oops thank you!

CoreyEWood · 2024-10-08T20:11:54Z

README.md


 With edge-only mode enabled for a detector, when you make requests to it, you will only receive answers from the edge model (regardless of the confidence). Additionally, note that no image queries submitted this way will show up in the web app or be used to train the model. This option should therefore only be used if you don't need the model to improve and only want fast answers from the edge model.

-If edge-only mode is enabled on a detector and the edge inference model for that detector is not available, attempting to send image queries to that detector will return a 500 error response.
+With edge-only inference enabled for a detector, when you make requests to it, you will only receive answers from the edge model (regardless of the confidence). However, image queries submitted this way with confidences below the threshold will be used to train the model. This option should be used when you want fast edge answers regardless of confidence but still want the model to improve.


Maybe to be more clear this should say something like, "However, when the edge model makes a prediction with confidence below the threshold, that image query will also be escalated to the cloud and used to train the model"?

CoreyEWood · 2024-10-08T20:19:58Z

test/core/test_configs.py

+from app.core.configs import DetectorConfig
+
+
+def test_detector_config():


Awesome to have this test, maybe it would be good to name it something more specific like "test_detector_config_with_both_edge_modes" in case we want to add more detector config tests in the future?

CoreyEWood · 2024-10-08T20:21:41Z

app/api/routes/image_queries.py

+            if edge_only_inference and not _is_confident_enough(
+                confidence=confidence,
+                confidence_threshold=confidence_threshold,
+            ):
+                logger.info("Escalating to the cloud API server for future training due to low confidence.")
+                background_tasks.add_task(safe_call_sdk, gl.ask_async, detector=detector_id, image=image)


This is great! Love how simple this is.

brandon-groundlight

Looks good to me. I do think deduping the images will become important in the near future, but for internal demos this should be great

…dge-endpoint into edge-with-training

f-wright and others added 30 commits October 4, 2024 15:50

initial edge + cloud changes

6e8b175

initial try for edge only inference

2aad26c

Merge branch 'main' into edge-with-training

111d565

add edge inference exclusion

8809c51

working background tasks

5a1a838

Automatically reformatting code with black and isort

38b7d5a

update readme

826c144

Merge branch 'edge-with-training' of /~https://github.com/groundlight/e…

23e78b7

…dge-endpoint into edge-with-training

updated validators

c7cfd54

update detector config only

1c8b7e6

adding testing

c2d3317

Automatically reformatting code with black and isort

4bf735b

remove edge only inference from test setup

893f1ef

Merge branch 'edge-with-training' of /~https://github.com/groundlight/e…

b1a9dc8

…dge-endpoint into edge-with-training

clean up

929456d

Merge branch 'main' into edge-with-training

f2531ed

remove det id

4ca87da

fix import

730b999

fix missing paren

49dbebc

update validator to run regardless of which field is set first

660f910

Automatically reformatting code with black and isort

b7ee52d

update validator for pydantic v2

6927e3a

gMerge branch 'edge-with-training' of /~https://github.com/groundlight/…

5475b05

…edge-endpoint into edge-with-training

Automatically reformatting code with black and isort

497bd77

switch to validation error

6768872

Merge branch 'edge-with-training' of /~https://github.com/groundlight/e…

735dab2

…dge-endpoint into edge-with-training

debugging test

f562ca9

skip testing setup for now

6bc667f

update to model validator and add back test

c580d35

update from Tylers changes

5d9688a

f-wright added 3 commits October 8, 2024 11:59

another try for the validator

957429b

fix config finally

0c611e4

Merge branch 'main' into edge-with-training

ac8ec71

f-wright marked this pull request as ready for review October 8, 2024 19:54

f-wright changed the title ~~Edge only inference + cloud training~~ Edge only inference with cloud training Oct 8, 2024

f-wright requested review from tyler-romero and CoreyEWood October 8, 2024 19:57

Automatically reformatting code with black and isort

60cf114

CoreyEWood approved these changes Oct 8, 2024

View reviewed changes

brandon-groundlight approved these changes Oct 8, 2024

View reviewed changes

f-wright added 2 commits October 8, 2024 13:45

PR review changes

53732f4

Merge branch 'edge-with-training' of /~https://github.com/groundlight/e…

551cf9a

…dge-endpoint into edge-with-training

f-wright merged commit 3755ba4 into main Oct 8, 2024
6 checks passed

f-wright deleted the edge-with-training branch October 8, 2024 23:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Edge only inference with cloud training #101

Edge only inference with cloud training #101

f-wright commented Oct 7, 2024 •

edited

Loading

CoreyEWood left a comment

CoreyEWood Oct 8, 2024 •

edited

Loading

f-wright Oct 8, 2024

CoreyEWood Oct 8, 2024

f-wright Oct 8, 2024

CoreyEWood Oct 8, 2024

CoreyEWood Oct 8, 2024

CoreyEWood Oct 8, 2024

brandon-groundlight left a comment

		from app.core.configs import DetectorConfig


		def test_detector_config():

Edge only inference with cloud training #101

Edge only inference with cloud training #101

Conversation

f-wright commented Oct 7, 2024 • edited Loading

CoreyEWood left a comment

Choose a reason for hiding this comment

CoreyEWood Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

f-wright Oct 8, 2024

Choose a reason for hiding this comment

CoreyEWood Oct 8, 2024

Choose a reason for hiding this comment

f-wright Oct 8, 2024

Choose a reason for hiding this comment

CoreyEWood Oct 8, 2024

Choose a reason for hiding this comment

CoreyEWood Oct 8, 2024

Choose a reason for hiding this comment

CoreyEWood Oct 8, 2024

Choose a reason for hiding this comment

brandon-groundlight left a comment

Choose a reason for hiding this comment

f-wright commented Oct 7, 2024 •

edited

Loading

CoreyEWood Oct 8, 2024 •

edited

Loading