Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][DAT-3394] Add support for maxcompute as source type (batch) #130

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

vinoth-gojek
Copy link

@vinoth-gojek vinoth-gojek commented Jan 14, 2025

This is WIP or draft MR.

The changes in MR are to enable MaxCompute as a datasource type for batch jobs (offline to online ingestion)

  1. The MR adds the BATCH_MAXCOMPUTE enum
  2. The changes in sdk enable clients to use maxcompute as source type
  3. The spark override is intentionally left out and I will add it in the upcoming MRs

With these changes, record with batch_source maxcompute is added to the datasources table

postgres=> select * from data_sources;
42 | BATCH_MAXCOMPUTE | {"table_ref":"datasource:playground.dsp_maxcompute_dogfood"} | {} | created_timestamp | created_timestamp |

@vinoth-gojek vinoth-gojek force-pushed the DAT-3394_add_MAXCOMPUTE_enum branch from f7c55f6 to bc0658b Compare January 16, 2025 06:44
@vinoth-gojek vinoth-gojek changed the title [WIP][DAT-3394] Add support for batch maxcompute [WIP][DAT-3394] Add support for maxcompute as source type (batch) Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant