Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add resource ID filtering in fetch augment-vis obj queries #4608

Merged
merged 3 commits into from
Jul 24, 2023

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Jul 21, 2023

Description

This PR provides optimizations to the saved object loader calls made in the vis embeddable. Specifically, when fetching the saved objects, it adds arguments to the query to filter by matching plugin resource IDs, when applicable. Before, all objects were returned and post-processing filtering was done. Now, we leverage OpenSearch to handle the filtering for us.

To make this change, a new optional argument has been added to the findAll() function to specify the search fields. Before, it was hardcoded to name and description. Now, we want to search on the pluginResource.id field such that we only return objects that match the input resource IDs.

This fix provides massive performance improvements when rendering the View Events flyout. Specifically, it prevents n^2 - n objects being returned, where n is the number of associations. This also means an equivalent amount of prevented plugin expression function runs, any queries ran within those expression functions, etc.

For example, suppose there are 10 associations to a visualization, and a user opens the View Events flyout.
Before, each visualization would be fetching all 10 augment-vis objects, executing all 10 expression functions (which would mean fetching all 10 resources, results for each of them, etc.). Each vis (10 executions) x 10 == 100 executions.
Now, each visualization only fetches the relevant augment-vis saved object, so only one expression function is ran. Each vis (1 execution) x 10 = 10 executions, a savings of 90%.

Before this change, the time to render the view events flyout as the number of visualizations/associations grew was exponential. Now, it is linear. Example of network request savings seen on my local:
Opening flyout with 5 associations = 98 requests (before) - 58 requests (after) = 40 requests prevented = ~41% less total requests.
10 associations = 288 requests (before) - 108 requests (after) = 180 requests prevented = ~63% less total requests

For more information on plugin resource IDs and how they are used, see "The VisAugmenterEmbeddableConfig" section in #3415

Quick results on my local (top row is association count)

|        | 10   | 20   | 40   | 50   |
|--------|------|------|------|------|
| before | 1.7s | 3.4s | 5.4s | 9.3s |
| after  | 1.3s | 2.0s | 2.5s | 2.9s |

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
    • yarn test:ftr
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
@codecov
Copy link

codecov bot commented Jul 21, 2023

Codecov Report

Merging #4608 (b391212) into main (adb538e) will increase coverage by 0.00%.
The diff coverage is 85.71%.

@@           Coverage Diff           @@
##             main    #4608   +/-   ##
=======================================
  Coverage   66.15%   66.16%           
=======================================
  Files        3314     3314           
  Lines       63910    63911    +1     
  Branches     9967     9970    +3     
=======================================
+ Hits        42281    42288    +7     
+ Misses      19184    19179    -5     
+ Partials     2445     2444    -1     
Flag Coverage Δ
Linux_1 34.74% <85.71%> (+0.01%) ⬆️
Linux_2 55.04% <ø> (ø)
Linux_3 43.17% <0.00%> (-0.01%) ⬇️
Linux_4 35.10% <100.00%> (+<0.01%) ⬆️
Windows_1 34.75% <85.71%> (+0.01%) ⬆️
Windows_2 55.00% <ø> (ø)
Windows_3 43.17% <0.00%> (-0.01%) ⬇️
Windows_4 35.10% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...izations/public/embeddable/visualize_embeddable.ts 19.65% <0.00%> (+0.33%) ⬆️
...objects/public/saved_object/saved_object_loader.ts 85.71% <100.00%> (+4.23%) ⬆️
...nter/public/saved_augment_vis/saved_augment_vis.ts 97.50% <100.00%> (ø)
src/plugins/vis_augmenter/public/utils/utils.ts 96.55% <100.00%> (+0.18%) ⬆️

... and 1 file with indirect coverage changes

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
@bandinib-amzn
Copy link
Member

Overall, this looks great! Could you please include unit tests (UT) to ensure quality?

Copy link
Member

@joshuarrrr joshuarrrr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ohltyler looks great! Just need to add the new param to the jsdocs comments.

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
@ohltyler
Copy link
Member Author

Overall, this looks great! Could you please include unit tests (UT) to ensure quality?

Added in latest commit - also enforces that the hasReference field is propagated as expected as well (from improvement in #4595) :)

@ohltyler
Copy link
Member Author

@bandinib-amzn @joshuarrrr all checks have passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants