[EXE-1528] Send jdbcOptions as MergedParameter to driver

[ShaoFuWu]

https://actioniq.atlassian.net/browse/EXE-1528
Didn't send queryGroup to driver, details here

sc.setLogLevel("info")
val redshiftOpts = Map(
  "aws_iam_role" -> "arn:aws:iam::938824509083:role/journeys-redshift-dev-redshift-role",
  "tempDir" -> s"s3a://aiq-1d-expire-dev/redshift/109",
  "driver" -> "com.amazon.redshift.jdbc42.Driver",
  "dbtable" -> "qahybridredshift.rd_dim_item",
  "queryGroup" -> "123456678908",
  "aiq_partner" -> "partnerName",
  "ApplicationName" -> "appName",
  "password" -> "",
  "user" -> "admin",
  "url" -> "jdbc:redshift://test-hybridcompute.938824509083.us-east-1.redshift-serverless.amazonaws.com:5439/dev?"
)
spark.read.format("io.github.spark_redshift_community.spark.redshift").options(redshiftOpts).load().show()

Sending a query from spark-shell and see the query_label `select * from SYS_QUERY_HISTORY where query_label = '123456678908'

Deploy Note

sbt publishM2       
aws s3 sync ~/.m2/repository/io/github/spark-redshift-community/ s3://aiq-artifacts/releases/io/github/spark-redshift-community

[yannistze]

Here you will log every key value, not just the user, password ones like in the original implementation 🤔 Do you need to be logging them for "Prod" or is this a testing remnant ? I am not sure we should be logging them cause they may contain creds 🤷‍♂️

[ShaoFuWu]

fuuuuuuck will remove, this is for my testing 🤦, good catch!

[yannistze]

Lgtm 👍

[MasterDDT]

I'm curious, did you look into the history of why they ignore all the other params?

Even on latest master, only use these 3 params and ignore the others /~https://github.com/spark-redshift-community/spark-redshift/blob/master/src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftRelation.scala#L111

[ShaoFuWu]

Looks like they were fixing the same invalid character 8 years ago like we were last week 😅 , the way they fix is passing user/password as part of the options in this pr

[MasterDDT]

Nice 👍 , this looks like an easy one you could contribute back to the open source fork?