Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove gke_command usages #4900

Merged
merged 1 commit into from
Feb 5, 2024
Merged

Remove gke_command usages #4900

merged 1 commit into from
Feb 5, 2024

Conversation

scholtzan
Copy link
Collaborator

@scholtzan scholtzan commented Jan 26, 2024 •
edited
Loading

Fixes #4895

We should wait with merging until next week and make those on Airflow Triage aware of these changes

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
  • If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
    in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
  • If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
  • When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

@dataops-ci-bot
Copy link

Integration report for "Remove gke_command usages"

sql.diff

Click to expand! 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_accounts_backend_external.py /tmp/workspace/generated-sql/dags/bqetl_accounts_backend_external.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_accounts_backend_external.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_accounts_backend_external.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_accounts_backend_external
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_contact_export.py /tmp/workspace/generated-sql/dags/bqetl_acoustic_contact_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_contact_export.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_acoustic_contact_export.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_acoustic_contact_export
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_raw_recipient_export.py /tmp/workspace/generated-sql/dags/bqetl_acoustic_raw_recipient_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_raw_recipient_export.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_acoustic_raw_recipient_export.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_acoustic_raw_recipient_export
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_activity_stream.py /tmp/workspace/generated-sql/dags/bqetl_activity_stream.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_activity_stream.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_activity_stream.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_activity_stream
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_addons.py /tmp/workspace/generated-sql/dags/bqetl_addons.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_addons.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_addons.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_addons
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_adjust.py /tmp/workspace/generated-sql/dags/bqetl_adjust.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_adjust.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_adjust.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_adjust
@@ -50,9 +51,9 @@
     doc_md=docs,
     tags=tags,
 ) as dag:
-    adjust_derived__adjust_cohort__v1 = gke_command(
+    adjust_derived__adjust_cohort__v1 = GKEPodOperator(
         task_id="adjust_derived__adjust_cohort__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/adjust_derived/adjust_cohort_v1/query.py",
         ]
@@ -64,14 +65,14 @@
             "--adjust_app_list",
             "{{ var.value.ADJUST_APP_TOKEN_LIST}}",
         ],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="mhirose@mozilla.com",
         email=["mhirose@mozilla.com", "telemetry-alerts@mozilla.com"],
     )
 
-    adjust_derived__adjust_deliverables__v1 = gke_command(
+    adjust_derived__adjust_deliverables__v1 = GKEPodOperator(
         task_id="adjust_derived__adjust_deliverables__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/adjust_derived/adjust_deliverables_v1/query.py",
         ]
@@ -83,7 +84,7 @@
             "--adjust_app_list",
             "{{ var.value.ADJUST_APP_TOKEN_LIST}}",
         ],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="mhirose@mozilla.com",
         email=["mhirose@mozilla.com", "telemetry-alerts@mozilla.com"],
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_ads.py /tmp/workspace/generated-sql/dags/bqetl_ads.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_ads.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_ads.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_ads
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_amo_stats.py /tmp/workspace/generated-sql/dags/bqetl_amo_stats.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_amo_stats.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_amo_stats.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_amo_stats
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_analytics_aggregations
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_analytics_tables
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_campaign_cost_breakdowns.py /tmp/workspace/generated-sql/dags/bqetl_campaign_cost_breakdowns.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_campaign_cost_breakdowns.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_campaign_cost_breakdowns.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_campaign_cost_breakdowns
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_cjms_nonprod.py /tmp/workspace/generated-sql/dags/bqetl_cjms_nonprod.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_cjms_nonprod.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_cjms_nonprod.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 from fivetran_provider.operators.fivetran import FivetranOperator
 from fivetran_provider.sensors.fivetran import FivetranSensor
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_core.py /tmp/workspace/generated-sql/dags/bqetl_core.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_core.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_core.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_core
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_crash.py /tmp/workspace/generated-sql/dags/bqetl_crash.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_crash.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_crash.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_crash
@@ -57,14 +58,14 @@
         depends_on_past=False,
     )
 
-    telemetry_derived__crash_signatures__v1 = gke_command(
+    telemetry_derived__crash_signatures__v1 = GKEPodOperator(
         task_id="telemetry_derived__crash_signatures__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/telemetry_derived/crash_signatures_v1/query.py",
         ]
         + ["--submission-date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="dthorn@mozilla.com",
         email=["dthorn@mozilla.com", "telemetry-alerts@mozilla.com"],
         retry_delay=datetime.timedelta(seconds=300),
@@ -82,14 +83,14 @@
         depends_on_past=False,
     )
 
-    telemetry_derived__crash_symbols__v1 = gke_command(
+    telemetry_derived__crash_symbols__v1 = GKEPodOperator(
         task_id="telemetry_derived__crash_symbols__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/telemetry_derived/crash_symbols_v1/query.py",
         ]
         + ["--submission-date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="dthorn@mozilla.com",
         email=["dthorn@mozilla.com", "telemetry-alerts@mozilla.com"],
         retry_delay=datetime.timedelta(seconds=300),
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_ctxsvc_derived.py /tmp/workspace/generated-sql/dags/bqetl_ctxsvc_derived.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_ctxsvc_derived.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_ctxsvc_derived.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_ctxsvc_derived
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_default.py /tmp/workspace/generated-sql/dags/bqetl_default.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_default.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_default.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_default
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_deletion_request_volume.py /tmp/workspace/generated-sql/dags/bqetl_deletion_request_volume.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_deletion_request_volume.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_deletion_request_volume.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_deletion_request_volume
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_desktop_funnel.py /tmp/workspace/generated-sql/dags/bqetl_desktop_funnel.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_desktop_funnel.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_desktop_funnel.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_desktop_funnel
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_desktop_installs_v1.py /tmp/workspace/generated-sql/dags/bqetl_desktop_installs_v1.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_desktop_installs_v1.py	2024-01-26 22:09:20.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_desktop_installs_v1.py	2024-01-26 22:11:18.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_desktop_installs_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_desktop_mobile_search_monthly.py /tmp/workspace/generated-sql/dags/bqetl_desktop_mobile_search_monthly.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_desktop_mobile_search_monthly.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_desktop_mobile_search_monthly.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_desktop_mobile_search_monthly
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_desktop_platform.py /tmp/workspace/generated-sql/dags/bqetl_desktop_platform.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_desktop_platform.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_desktop_platform.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_desktop_platform
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_devtools.py /tmp/workspace/generated-sql/dags/bqetl_devtools.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_devtools.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_devtools.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_devtools
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_domain_meta.py /tmp/workspace/generated-sql/dags/bqetl_domain_meta.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_domain_meta.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_domain_meta.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_domain_meta
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_download_funnel_attribution.py /tmp/workspace/generated-sql/dags/bqetl_download_funnel_attribution.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_download_funnel_attribution.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_download_funnel_attribution.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_download_funnel_attribution
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_error_aggregates.py /tmp/workspace/generated-sql/dags/bqetl_error_aggregates.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_error_aggregates.py	2024-01-26 22:09:15.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_error_aggregates.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_error_aggregates
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_event_rollup.py /tmp/workspace/generated-sql/dags/bqetl_event_rollup.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_event_rollup.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_event_rollup.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_event_rollup
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_experimenter_experiments_import.py /tmp/workspace/generated-sql/dags/bqetl_experimenter_experiments_import.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_experimenter_experiments_import.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_experimenter_experiments_import.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_experimenter_experiments_import
@@ -52,14 +53,14 @@
     doc_md=docs,
     tags=tags,
 ) as dag:
-    monitoring__experimenter_experiments__v1 = gke_command(
+    monitoring__experimenter_experiments__v1 = GKEPodOperator(
         task_id="monitoring__experimenter_experiments__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-experiments/monitoring/experimenter_experiments_v1/query.py",
         ]
         + [],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_experiments_daily.py /tmp/workspace/generated-sql/dags/bqetl_experiments_daily.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_experiments_daily.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_experiments_daily.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_experiments_daily
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_feature_usage.py /tmp/workspace/generated-sql/dags/bqetl_feature_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_feature_usage.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_feature_usage.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_feature_usage
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fenix_event_rollup.py /tmp/workspace/generated-sql/dags/bqetl_fenix_event_rollup.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fenix_event_rollup.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fenix_event_rollup.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fenix_event_rollup
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fenix_external.py /tmp/workspace/generated-sql/dags/bqetl_fenix_external.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fenix_external.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fenix_external.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fenix_external
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_firefox_ios.py /tmp/workspace/generated-sql/dags/bqetl_firefox_ios.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_firefox_ios.py	2024-01-26 22:09:18.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_firefox_ios.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_firefox_ios
@@ -355,14 +356,14 @@
         depends_on_past=True,
     )
 
-    org_mozilla_ios_firefox__unified_metrics__v1 = gke_command(
+    org_mozilla_ios_firefox__unified_metrics__v1 = GKEPodOperator(
         task_id="org_mozilla_ios_firefox__unified_metrics__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/org_mozilla_ios_firefox/unified_metrics_v1/query.py",
         ]
         + [],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="kik@mozilla.com",
         email=["frank@mozilla.com", "kik@mozilla.com", "telemetry-alerts@mozilla.com"],
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_apple_ads.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_apple_ads.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_apple_ads.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_apple_ads.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fivetran_apple_ads
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_copied_tables.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_copied_tables.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_copied_tables.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_copied_tables.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fivetran_copied_tables
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_costs.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_costs.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_costs.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_costs.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 from fivetran_provider.operators.fivetran import FivetranOperator
 from fivetran_provider.sensors.fivetran import FivetranSensor
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_google_ads.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_google_ads.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_google_ads.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_google_ads.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fivetran_google_ads
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fog_decision_support.py /tmp/workspace/generated-sql/dags/bqetl_fog_decision_support.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fog_decision_support.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fog_decision_support.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fog_decision_support
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fxa_events.py /tmp/workspace/generated-sql/dags/bqetl_fxa_events.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fxa_events.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fxa_events.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_fxa_events
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_generated_funnels
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glam_export.py /tmp/workspace/generated-sql/dags/bqetl_glam_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glam_export.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glam_export.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_glam_export
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glam_refresh_aggregates.py /tmp/workspace/generated-sql/dags/bqetl_glam_refresh_aggregates.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glam_refresh_aggregates.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glam_refresh_aggregates.py	2024-01-26 22:11:18.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_glam_refresh_aggregates
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py	2024-01-26 22:09:22.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py	2024-01-26 22:11:20.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_glean_usage
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_google_analytics_derived.py /tmp/workspace/generated-sql/dags/bqetl_google_analytics_derived.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_google_analytics_derived.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_google_analytics_derived.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_google_analytics_derived
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_google_analytics_derived_ga4.py /tmp/workspace/generated-sql/dags/bqetl_google_analytics_derived_ga4.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_google_analytics_derived_ga4.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_google_analytics_derived_ga4.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_google_analytics_derived_ga4
@@ -48,17 +49,6 @@
     doc_md=docs,
     tags=tags,
 ) as dag:
-    ga_derived__www_site_downloads__v2 = bigquery_etl_query(
-        task_id="ga_derived__www_site_downloads__v2",
-        destination_table="www_site_downloads_v2",
-        dataset_id="ga_derived",
-        project_id="moz-fx-data-marketing-prod",
-        owner="kwindau@mozilla.com",
-        email=["kwindau@mozilla.com", "telemetry-alerts@mozilla.com"],
-        date_partition_parameter="submission_date",
-        depends_on_past=False,
-    )
-
     ga_derived__www_site_hits__v2 = bigquery_etl_query(
         task_id="ga_derived__www_site_hits__v2",
         destination_table="www_site_hits_v2",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_gud.py /tmp/workspace/generated-sql/dags/bqetl_gud.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_gud.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_gud.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_gud
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_internal_tooling.py /tmp/workspace/generated-sql/dags/bqetl_internal_tooling.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_internal_tooling.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_internal_tooling.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_internal_tooling
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_internet_outages.py /tmp/workspace/generated-sql/dags/bqetl_internet_outages.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_internet_outages.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_internet_outages.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_internet_outages
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_iprospect.py /tmp/workspace/generated-sql/dags/bqetl_iprospect.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_iprospect.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_iprospect.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_iprospect
@@ -49,14 +50,14 @@
     doc_md=docs,
     tags=tags,
 ) as dag:
-    iprospect__adspend__v1 = gke_command(
+    iprospect__adspend__v1 = GKEPodOperator(
         task_id="iprospect__adspend__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-marketing-prod/iprospect/adspend_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com", "echo@mozilla.com", "shong@mozilla.com"],
     )
@@ -72,14 +73,14 @@
         depends_on_past=True,
     )
 
-    iprospect__adspend_raw__v1 = gke_command(
+    iprospect__adspend_raw__v1 = GKEPodOperator(
         task_id="iprospect__adspend_raw__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-marketing-prod/iprospect/adspend_raw_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com", "echo@mozilla.com", "shong@mozilla.com"],
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_kpis_shredder.py /tmp/workspace/generated-sql/dags/bqetl_kpis_shredder.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_kpis_shredder.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_kpis_shredder.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_kpis_shredder
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_main_summary.py /tmp/workspace/generated-sql/dags/bqetl_main_summary.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_main_summary.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_main_summary.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_main_summary
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mdn_yari.py /tmp/workspace/generated-sql/dags/bqetl_mdn_yari.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mdn_yari.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mdn_yari.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mdn_yari
@@ -56,14 +57,14 @@
     doc_md=docs,
     tags=tags,
 ) as dag:
-    mdn_yari_derived__mdn_popularities__v1 = gke_command(
+    mdn_yari_derived__mdn_popularities__v1 = GKEPodOperator(
         task_id="mdn_yari_derived__mdn_popularities__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/mdn_yari_derived/mdn_popularities_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="fmerz@mozilla.com",
         email=[
             "fmerz@mozilla.com",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_messaging_system.py /tmp/workspace/generated-sql/dags/bqetl_messaging_system.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_messaging_system.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_messaging_system.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_messaging_system
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mobile_activation.py /tmp/workspace/generated-sql/dags/bqetl_mobile_activation.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mobile_activation.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mobile_activation.py	2024-01-26 22:11:16.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mobile_activation
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mobile_feature_usage.py /tmp/workspace/generated-sql/dags/bqetl_mobile_feature_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mobile_feature_usage.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mobile_feature_usage.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mobile_feature_usage
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mobile_search.py /tmp/workspace/generated-sql/dags/bqetl_mobile_search.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mobile_search.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mobile_search.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mobile_search
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_monitoring.py /tmp/workspace/generated-sql/dags/bqetl_monitoring.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_monitoring.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_monitoring.py	2024-01-26 22:11:15.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_monitoring
@@ -62,86 +63,86 @@
         task_concurrency=1,
     )
 
-    monitoring_derived__average_ping_sizes__v1 = gke_command(
+    monitoring_derived__average_ping_sizes__v1 = GKEPodOperator(
         task_id="monitoring_derived__average_ping_sizes__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/average_ping_sizes_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_etl_scheduled_queries_cost__v1 = gke_command(
+    monitoring_derived__bigquery_etl_scheduled_queries_cost__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_etl_scheduled_queries_cost__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_queries_cost_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_etl_scheduled_query_usage__v1 = gke_command(
+    monitoring_derived__bigquery_etl_scheduled_query_usage__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_etl_scheduled_query_usage__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_query_usage_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_table_storage__v1 = gke_command(
+    monitoring_derived__bigquery_table_storage__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_table_storage__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_v1/query.py",
         ]
         + [],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="wichan@mozilla.com",
         email=["ascholtz@mozilla.com", "wichan@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_table_storage_timeline_daily__v1 = gke_command(
+    monitoring_derived__bigquery_table_storage_timeline_daily__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_table_storage_timeline_daily__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_table_storage_timeline_daily_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="wichan@mozilla.com",
         email=["ascholtz@mozilla.com", "wichan@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_tables_inventory__v1 = gke_command(
+    monitoring_derived__bigquery_tables_inventory__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_tables_inventory__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_tables_inventory_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="wichan@mozilla.com",
         email=["ascholtz@mozilla.com", "wichan@mozilla.com"],
     )
 
-    monitoring_derived__bigquery_usage__v1 = gke_command(
+    monitoring_derived__bigquery_usage__v1 = GKEPodOperator(
         task_id="monitoring_derived__bigquery_usage__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_usage_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="wichan@mozilla.com",
         email=["ascholtz@mozilla.com", "wichan@mozilla.com"],
     )
@@ -157,14 +158,14 @@
         depends_on_past=False,
     )
 
-    monitoring_derived__column_size__v1 = gke_command(
+    monitoring_derived__column_size__v1 = GKEPodOperator(
         task_id="monitoring_derived__column_size__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/column_size_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
@@ -204,14 +205,14 @@
         depends_on_past=False,
     )
 
-    monitoring_derived__jobs_by_organization__v1 = gke_command(
+    monitoring_derived__jobs_by_organization__v1 = GKEPodOperator(
         task_id="monitoring_derived__jobs_by_organization__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/jobs_by_organization_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="mhirose@mozilla.com",
         email=["ascholtz@mozilla.com", "mhirose@mozilla.com"],
     )
@@ -227,9 +228,9 @@
         depends_on_past=False,
     )
 
-    monitoring_derived__shredder_rows_deleted__v1 = gke_command(
+    monitoring_derived__shredder_rows_deleted__v1 = GKEPodOperator(
         task_id="monitoring_derived__shredder_rows_deleted__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/shredder_rows_deleted_v1/query.py",
         ]
@@ -239,19 +240,19 @@
             "--destination_table",
             "moz-fx-data-shared-prod.monitoring_derived.shredder_rows_deleted_v1${{ds_nodash}}",
         ],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="akomar@mozilla.com",
         email=["akomar@mozilla.com", "ascholtz@mozilla.com"],
     )
 
-    monitoring_derived__stable_and_derived_table_sizes__v1 = gke_command(
+    monitoring_derived__stable_and_derived_table_sizes__v1 = GKEPodOperator(
         task_id="monitoring_derived__stable_and_derived_table_sizes__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/stable_and_derived_table_sizes_v1/query.py",
         ]
         + ["--date", "{{ macros.ds_add(ds, -1) }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="ascholtz@mozilla.com",
         email=["ascholtz@mozilla.com"],
     )
@@ -282,14 +283,14 @@
         sql_file_path="sql/moz-fx-data-shared-prod/monitoring_derived/structured_error_counts_v2/script.sql",
     )
 
-    monitoring_derived__structured_missing_columns__v1 = gke_command(
+    monitoring_derived__structured_missing_columns__v1 = GKEPodOperator(
         task_id="monitoring_derived__structured_missing_columns__v1",
-        command=[
+        arguments=[
             "python",
             "sql/moz-fx-data-shared-prod/monitoring_derived/structured_missing_columns_v1/query.py",
         ]
         + ["--date", "{{ ds }}"],
-        docker_image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
+        image="gcr.io/moz-fx-data-airflow-prod-88e0/bigquery-etl:latest",
         owner="amiyaguchi@mozilla.com",
         email=["amiyaguchi@mozilla.com", "ascholtz@mozilla.com"],
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_monitoring_airflow.py /tmp/workspace/generated-sql/dags/bqetl_monitoring_airflow.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_monitoring_airflow.py	2024-01-26 22:09:17.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_monitoring_airflow.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 from fivetran_provider.operators.fivetran import FivetranOperator
 from fivetran_provider.sensors.fivetran import FivetranSensor
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_org_derived.py /tmp/workspace/generated-sql/dags/bqetl_mozilla_org_derived.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_org_derived.py	2024-01-26 22:09:20.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mozilla_org_derived.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mozilla_org_derived
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_vpn_site_metrics.py /tmp/workspace/generated-sql/dags/bqetl_mozilla_vpn_site_metrics.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mozilla_vpn_site_metrics.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mozilla_vpn_site_metrics.py	2024-01-26 22:11:13.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_mozilla_vpn_site_metrics
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_newtab.py /tmp/workspace/generated-sql/dags/bqetl_newtab.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_newtab.py	2024-01-26 22:09:19.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_newtab.py	2024-01-26 22:11:17.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import GKEPodOperator
 from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, gke_command, bigquery_dq_check
+from utils.gcp import bigquery_etl_query, bigquery_dq_check
 
 docs = """
 ### bqetl_newtab
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_nondesktop.py /tmp/workspace/generated-sql/dags/bqetl_nondesktop.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_nondesktop.py	2024-01-26 22:09:16.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_nondesktop.py	2024-01-26 22:11:14.000000000 +0000
@@ -5,8 +5,9 @@
 from airflow.sensors.external_task import ExternalTaskSensor
 from airflow.utils.task_group import TaskGroup
 import datetime
+from operators.gcp_container_operator import

⚠️ Only part of the diff is displayed.

Link to full diff

@scholtzan scholtzan merged commit a4c7b0a into main Feb 5, 2024
19 checks passed
@scholtzan scholtzan deleted the remove-gke-command branch February 5, 2024 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kwindau kwindau kwindau approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Replace use of gke_command with GKEPodOperator in DAG jinja templates
3 participants
@scholtzan @dataops-ci-bot @kwindau