[EXE-1558] Setup build and deploy; add pushdown

[dorisZ017]

• Upgrade sbt version and plugins; remove unused plugins;
• Upgrade jdbc driver and hadoop version;
• Upgrade to java 17; fix issues with test dependencies and compiling;
• Setup deploying to jfrog
• Bringing in AIQ tracing and pushdowns in [EXE-1273] Extra pushdown & Unload ActionIQ-OSS/spark-redshift#9 and [EXE-1528] Send jdbcOptions as MergedParameter to driver ActionIQ-OSS/spark-redshift#10, also making them compatible with spark 3
• Exposing redshift statement in pushdown plan

Noticed some issue with partially invalid queries, documented here: https://actioniq.atlassian.net/browse/EXE-1614, will continue digging as part of bringing generic jdbc pushdown to flame.

Tested reading from redshift in /~https://github.com/ActionIQ/flame/pull/440/files

sc.setLogLevel("info")
// spark.conf.set("spark.aiq.sql.enable_jdbc_pushdown", true)
val redshiftOpts = Map(
  "aws_iam_role" -> "arn:aws:iam::938824509083:role/journeys-redshift-dev-redshift-role",
  "tempDir" -> s"s3a://aiq-1d-expire-dev/redshift/109",
  "driver" -> "com.amazon.redshift.jdbc42.Driver",
  "dbtable" -> "qahybridredshift.rd_dim_item",
  "queryGroup" -> "123456678908",
  "aiq_partner" -> "partnerName",
  "ApplicationName" -> "appName",
  "password" -> <>
  "user" -> <>,
  "url" -> "jdbc:redshift://test-hybridcompute.938824509083.us-east-1.redshift-serverless.amazonaws.com:5439/dev?"
)
val df = spark.read.format("io.github.spark_redshift_community.spark.redshift").options(redshiftOpts).load()
df.createOrReplaceTempView("t")
val qe = spark.sql("select color from t where lower(color) = 'snow'").queryExecution
qe.sparkPlan.asInstanceOf[io.github.spark_redshift_community.spark.redshift.pushdowns.RedshiftPushDownPlan].statement
res8: String = SELECT ( SUBQUERY_1.color ) AS SUBQUERY_2_COL_0 FROM ( SELECT * FROM ( SELECT * FROM qahybridredshift.rd_dim_item AS REDSHIFT_QUERY_ALIAS ) AS SUBQUERY_0 WHERE ( ( SUBQUERY_0.color IS NOT NULL ) AND ( LOWER ( SUBQUERY_0.color ) = ?(Some(snow)) ) ) ) AS SUBQUERY_1

sbt test
sbt publish

[MasterDDT]

Hmm how did Github let you request me for review if its still in draft mode 😄

[dorisZ017]

Hmm how did Github let you request me for review if its still in draft mode 😄

No I was just adding the build and publish stuff but later figured I could just add pushdowns on top of it so converted it to draft... I split the pushdowns to another PR if you want to review the build part first

[ShaoFuWu]

looks good 👍 one thing to follow up, can you test the pushdowns that we listed in our connector confluence and make sure those still work in spark 3, I guess can wait until you deploy spark and run from AIQ if thats easier.
Also, can you please fix the formatting in RedshiftPushDownQuery.scala please.

[MasterDDT]

These dont work with java 17?

[dorisZ017]

They were removed in java 17 (or earlier): https://docs.oracle.com/en/java/javase/17/docs/specs/man/java.html#removed-java-options

[MasterDDT]

We use aws java 1.12.x, upgrade this one? Or is it just for tests and we dont care

[dorisZ017]

I was referencing the dependencies of hadoop-aws: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.2 but yeah I try upgrading here

[MasterDDT]

Prob dont need this anymore 🤷 , now that your exposed the statement in the plan. But I guess its fine to keep.

I dont see it used below in any tests?

[dorisZ017]

Will try adding some unit tests later