Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BigQuery related integration tests to make unique BQ datasets and tables #1760

Closed
adchia opened this issue Aug 4, 2021 · 1 comment · Fixed by #1801 or #1818
Closed

Fix BigQuery related integration tests to make unique BQ datasets and tables #1760

adchia opened this issue Aug 4, 2021 · 1 comment · Fixed by #1801 or #1818
Labels

Comments

@adchia
Copy link
Collaborator

adchia commented Aug 4, 2021

Currently, for integration tests, we try to "recreate/reupdate" the same "ds" or "test_ingestion" dataset in BigQuery. This can cause issues though if e.g. the test runner isn't an admin privilege because in particular updating the timestamp of the table requires admin privileges.

See test_offline_online_store_consistency.py and data_source_utils.py

This is to track creating unique bq datasets with each test run to ensure integration tests can pass without managing table permissions in BQ.

@adchia adchia added the good first issue Good for newcomers label Aug 4, 2021
@adchia adchia changed the title Fix BigQuery related integration tests to make unique BQ tables Fix BigQuery related integration tests to make unique BQ datasets Aug 4, 2021
@woop woop changed the title Fix BigQuery related integration tests to make unique BQ datasets Fix BigQuery related integration tests to make unique BQ datasets and tables Aug 24, 2021
@woop
Copy link
Member

woop commented Aug 24, 2021

Related to this issue is /~https://github.com/feast-dev/feast/runs/3403674874?check_suite_focus=true. We are hitting GCP rate limits because we've hardcoded the table name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants