Experimental transaction support

[zhicwu]

No description provided.

[cevenrokt]

@zhicwu this does not work in my case, regardless of whether I specify custom_http_headers or not. I use https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.3.2-patch11/clickhouse-jdbc-0.3.2-patch11-all.jar
and my clickhouse cluster version is 22.8.4.7

This is my write statement:

df.write \
    .format('jdbc') \
    .option('driver', 'com.clickhouse.jdbc.ClickHouseDriver') \
    .option('ssl', 'true') \
    .option('url', clickhouse_url) \
    .option('custom_http_headers', 'implicit_transaction') \
    .option('dbtable', 'db.table') \
    .mode('append') \
    .save()

I am still getting the following errors and transaction ends up being aborted and rolled back:

22/11/04 00:59:21 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction [74d2332b-72a9-4c08-af0a-4f7f1f82a48e] (2 queries & 0 savepoints) is committed.
22/11/04 00:59:21 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction is not supported. You may change jdbcCompliant to false to throw SQLException instead.
22/11/04 00:59:21 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction [2cf0939f-5ee4-4c6a-b9c5-6da164458fa6] (0 queries & 0 savepoints) is committed.
22/11/04 00:59:25 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction is not supported. You may change jdbcCompliant to false to throw SQLException instead.
22/11/04 00:59:25 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction [1d57a83e-ce70-4019-8fab-f3461eb305b9] (1 queries & 0 savepoints) is rolled back.
2/11/04 00:59:25 WARN ClickHouseConnectionImpl: [JDBC Compliant Mode] Transaction [2d59027b-cf0d-4596-8031-df98639c55d1] (0 queries & 0 savepoints) is committed.
22/11/04 00:59:25 ERROR Executor: Exception in task 11.0 in stage 40.0 (TID 660)
java.sql.SQLException: connect timed out, server ClickHouseNode [uri=https://clickhouse-cluster:8123/default, options={driver=com.clickhouse.jdbc.ClickHouseDriver,dbtable=db.table,custom_http_headers=implicit_transaction,url=jdbc:clickhouse://clickhouse-cluster:8123}]@-2124678654

[genzgd]

@cevenrokt

I got this working in Scala Spark, I assume the approach is similar for PySpark:

  df.write
    .format("jdbc")
    .option("driver", "com.clickhouse.jdbc.ClickHouseDriver")
    .option("transactionSupport", true)
    .option("url", "jdbc:clickhouse:https://<CLICKHOUSE_HOST>:8443")
    .option("dbtable", "df_test")
    .option("user", "default")
    .option("password", "<CLICKHOUSE_PASSWORD>")
    .mode("append")
    .save()

Note that the ssl option is not necessary; for HTTPS/TLS it looks like you need https and the https port (8443 is the default) in the url option. Also I believe that the transactionSupport option automatically adds the implicit_transaction http header, so a separate option for custom_http_headers should not be necessary.

Note that for transactions to work, the top level setting

<allow_experimental_transactions>1</allow_experimental_transactions>

must be configured in your ClickHouse server config.xml (or a file included from the config.d subdirectory).

In addition, experimental transactions require that your ClickHouse server have a zookeeper connection, and they do not work on ReplicatedMergeTrees. Accordingly, for the time being we don't recommend enabling them for production workloads.