Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom data sources #1713

Merged
merged 17 commits into from
Jul 17, 2021
Merged

Conversation

achals
Copy link
Member

@achals achals commented Jul 15, 2021

What this PR does / why we need it:

After allowing for custom online stores and offline stores, we need to update the DataSource API so that implementors can create custom. DataSources to be used in their feature repo, and provide a mechanism through which these DataSources can be serialized/deserialized from the registry.

This is done by introducing a super simple CustomSourceOptions message that contains information about the class used for the source, and any serialized contents. Implementors are responsible for defining a to_proto and from_proto method and they can serialize the contents into this class as they need.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Allow for custom Data Sources to be defined.

@achals achals requested review from tsotnet, woop and a team as code owners July 15, 2021 17:08
@achals achals changed the title Achal/custom data sources Add custom data sources Jul 15, 2021
achals added 6 commits July 15, 2021 13:04
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
…a_source.py

Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
@achals achals force-pushed the achal/custom-data-sources branch from fba75cb to 6679e05 Compare July 15, 2021 20:07
Signed-off-by: Achal Shah <achals@gmail.com>
@codecov-commenter
Copy link

codecov-commenter commented Jul 15, 2021

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 79.47883% with 63 lines in your changes missing coverage. Please review.

Project coverage is 84.47%. Comparing base (c5300cb) to head (9e3f4e3).
Report is 1737 commits behind head on master.

Files with missing lines Patch % Lines
sdk/python/feast/infra/offline_stores/redshift.py 67.44% 28 Missing ⚠️
sdk/python/feast/infra/offline_stores/bigquery.py 80.23% 17 Missing ⚠️
sdk/python/feast/data_source.py 74.19% 8 Missing ⚠️
sdk/python/feast/infra/offline_stores/file.py 88.57% 8 Missing ⚠️
sdk/python/feast/feature_view.py 66.66% 2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1713      +/-   ##
==========================================
+ Coverage   84.45%   84.47%   +0.01%     
==========================================
  Files          79       79              
  Lines        7012     7071      +59     
==========================================
+ Hits         5922     5973      +51     
- Misses       1090     1098       +8     
Flag Coverage Δ
integrationtests 84.40% <79.47%> (+0.01%) ⬆️
unittests 69.45% <62.54%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

achals added 5 commits July 15, 2021 13:35
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
achals added 2 commits July 16, 2021 12:08
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop
Copy link
Member

woop commented Jul 16, 2021

Just waiting for merge conflict to be resolved. Otherwise lgtm.

achals added 3 commits July 16, 2021 18:51
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants