[jdbc] Add support for case sensitive table names reflecting item nam…

…es 1:1 (openhab#13544) * Do not append number when using real item names * Extract getTableName to separate class * Add initial test coverage * Extract migration logic to separate class * Support migration from real names back to numbered * Simplify zero-padding * Fix NullPointerException * Fix MySQL compatibility when CLIENT_MULTI_STATEMENTS option is not set * Add option for case sensitive table names * Add real name with suffix mode for backwards compatibility * Remove real name in lower case without suffix mode * Map directly from item name to table name * Fix ambiguous table name scenario * Add additional testcase * Add migration path for changed table prefix * Drop items table when using direct mapping * Add configuration note * Fix table alignment * Extend description as more migration paths are now supported * Do not stop halfway through a migration * For clarity, do not use abbreviation for operating system Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
nemerdaud · Feb 28, 2023 · c34ad87 · c34ad87
1 parent 24c1abf
commit c34ad87
Show file tree

Hide file tree

Showing 13 changed files with 780 additions and 173 deletions.
diff --git a/bundles/org.openhab.persistence.jdbc/README.md b/bundles/org.openhab.persistence.jdbc/README.md
@@ -38,38 +38,39 @@ The following databases are currently supported and tested:
 
 This service can be configured in the file `services/jdbc.cfg`.
 
-| Property                  | Default                                                      | Required  | Description                                                  |
-| ------------------------- | ------------------------------------------------------------ | :-------: | ------------------------------------------------------------ |
-| url                       |                                                              |    Yes    | JDBC URL to establish a connection to your database.  Examples:<br/><br/>`jdbc:derby:./testDerby;create=true`<br/>`jdbc:h2:./testH2`<br/>`jdbc:hsqldb:./testHsqlDb`<br/>`jdbc:mariadb://192.168.0.1:3306/testMariadb`<br/>`jdbc:mysql://192.168.0.1:3306/testMysql?serverTimezone=UTC`<br/>`jdbc:postgresql://192.168.0.1:5432/testPostgresql`<br/>`jdbc:timescaledb://192.168.0.1:5432/testPostgresql`<br/>`jdbc:sqlite:./testSqlite.db`.<br/><br/>If no database is available it will be created; for example the url `jdbc:h2:./testH2` creates a new H2 database in openHAB folder. Example to create your own MySQL database directly:<br/><br/>`CREATE DATABASE 'yourDB' CHARACTER SET utf8 COLLATE utf8_general_ci;` |
-| user                      |                                                              | if needed | database user name                                           |
-| password                  |                                                              | if needed | database user password                                       |
-| errReconnectThreshold     | 0                                                            |    No     | when the service is deactivated (0 means ignore)             |
-| sqltype.CALL              | `VARCHAR(200)`                                               |    No     | All `sqlType` options allow you to change the SQL data type used to store values for different openHAB item states.  See the following links for further information: [mybatis](https://mybatis.github.io/mybatis-3/apidocs/reference/org/apache/ibatis/type/JdbcType.html) [H2](https://www.h2database.com/html/datatypes.html) [PostgresSQL](https://www.postgresql.org/docs/9.3/static/datatype.html) |
-| sqltype.COLOR             | `VARCHAR(70)`                                                |    No     | see above                                                    |
-| sqltype.CONTACT           | `VARCHAR(6)`                                                 |    No     | see above                                                    |
-| sqltype.DATETIME          | `DATETIME`                                                   |    No     | see above                                                    |
-| sqltype.DIMMER            | `TINYINT`                                                    |    No     | see above                                                    |
-| sqltype.IMAGE             | `VARCHAR(65500)`                                             |    No     | see above                                                    |
-| sqltype.LOCATION          | `VARCHAR(50)`                                                |    No     | see above                                                    |
-| sqltype.NUMBER            | `DOUBLE`                                                     |    No     | see above                                                    |
-| sqltype.PLAYER            | `VARCHAR(20)`                                                |    No     | see above                                                    |
-| sqltype.ROLLERSHUTTER     | `TINYINT`                                                    |    No     | see above                                                    |
-| sqltype.STRING            | `VARCHAR(65500)`                                             |    No     | see above                                                    |
-| sqltype.SWITCH            | `VARCHAR(6)`                                                 |    No     | see above                                                    |
-| sqltype.tablePrimaryKey   | `TIMESTAMP`                                                  |    No     | type of `time` column for newly created item tables          |
-| sqltype.tablePrimaryValue | `NOW()`                                                      |    No     | value of `time` column for newly inserted rows               |
-| numberDecimalcount        | 3                                                            |    No     | for Itemtype "Number" default decimal digit count            |
-| tableNamePrefix           | `item`                                                       |    No     | table name prefix. For Migration from MySQL Persistence, set to `Item`. |
-| tableUseRealItemNames     | `false`                                                      |    No     | table name prefix generation.  When set to `true`, real item names are used for table names and `tableNamePrefix` is ignored.  When set to `false`, the `tableNamePrefix` is used to generate table names with sequential numbers. |
-| tableIdDigitCount         | 4                                                            |    No     | when `tableUseRealItemNames` is `false` and thus table names are generated sequentially, this controls how many zero-padded digits are used in the table name.  With the default of 4, the first table name will end with `0001`. For migration from the MySQL persistence service, set this to 0. |
-| rebuildTableNames         | false                                                        |    No     | rename existing tables using `tableUseRealItemNames` and `tableIdDigitCount`. USE WITH CARE! Deactivate after Renaming is done! |
-| jdbc.maximumPoolSize      | configured per database in package `org.openhab.persistence.jdbc.db.*` |    No     | Some embedded databases can handle only one connection. See [this link](/~https://github.com/brettwooldridge/HikariCP/issues/256) for more information |
-| jdbc.minimumIdle          | see above                                                    |    No     | see above                                                    |
-| enableLogTime             | `false`                                                      |    No     | timekeeping                                                  |
+| Property                    | Default                                                      | Required  | Description                                                  |
+| --------------------------- | ------------------------------------------------------------ | :-------: | ------------------------------------------------------------ |
+| url                         |                                                              |    Yes    | JDBC URL to establish a connection to your database.  Examples:<br/><br/>`jdbc:derby:./testDerby;create=true`<br/>`jdbc:h2:./testH2`<br/>`jdbc:hsqldb:./testHsqlDb`<br/>`jdbc:mariadb://192.168.0.1:3306/testMariadb`<br/>`jdbc:mysql://192.168.0.1:3306/testMysql?serverTimezone=UTC`<br/>`jdbc:postgresql://192.168.0.1:5432/testPostgresql`<br/>`jdbc:timescaledb://192.168.0.1:5432/testPostgresql`<br/>`jdbc:sqlite:./testSqlite.db`.<br/><br/>If no database is available it will be created; for example the url `jdbc:h2:./testH2` creates a new H2 database in openHAB folder. Example to create your own MySQL database directly:<br/><br/>`CREATE DATABASE 'yourDB' CHARACTER SET utf8 COLLATE utf8_general_ci;` |
+| user                        |                                                              | if needed | database user name                                           |
+| password                    |                                                              | if needed | database user password                                       |
+| errReconnectThreshold       | 0                                                            |    No     | when the service is deactivated (0 means ignore)             |
+| sqltype.CALL                | `VARCHAR(200)`                                               |    No     | All `sqlType` options allow you to change the SQL data type used to store values for different openHAB item states.  See the following links for further information: [mybatis](https://mybatis.github.io/mybatis-3/apidocs/reference/org/apache/ibatis/type/JdbcType.html) [H2](https://www.h2database.com/html/datatypes.html) [PostgresSQL](https://www.postgresql.org/docs/9.3/static/datatype.html) |
+| sqltype.COLOR               | `VARCHAR(70)`                                                |    No     | see above                                                    |
+| sqltype.CONTACT             | `VARCHAR(6)`                                                 |    No     | see above                                                    |
+| sqltype.DATETIME            | `DATETIME`                                                   |    No     | see above                                                    |
+| sqltype.DIMMER              | `TINYINT`                                                    |    No     | see above                                                    |
+| sqltype.IMAGE               | `VARCHAR(65500)`                                             |    No     | see above                                                    |
+| sqltype.LOCATION            | `VARCHAR(50)`                                                |    No     | see above                                                    |
+| sqltype.NUMBER              | `DOUBLE`                                                     |    No     | see above                                                    |
+| sqltype.PLAYER              | `VARCHAR(20)`                                                |    No     | see above                                                    |
+| sqltype.ROLLERSHUTTER       | `TINYINT`                                                    |    No     | see above                                                    |
+| sqltype.STRING              | `VARCHAR(65500)`                                             |    No     | see above                                                    |
+| sqltype.SWITCH              | `VARCHAR(6)`                                                 |    No     | see above                                                    |
+| sqltype.tablePrimaryKey     | `TIMESTAMP`                                                  |    No     | type of `time` column for newly created item tables          |
+| sqltype.tablePrimaryValue   | `NOW()`                                                      |    No     | value of `time` column for newly inserted rows               |
+| numberDecimalcount          | 3                                                            |    No     | for Itemtype "Number" default decimal digit count            |
+| tableNamePrefix             | `item`                                                       |    No     | table name prefix. For Migration from MySQL Persistence, set to `Item`. |
+| tableUseRealItemNames       | `false`                                                      |    No     | table name prefix generation.  When set to `true`, real item names are used for table names and `tableNamePrefix` is ignored.  When set to `false`, the `tableNamePrefix` is used to generate table names with sequential numbers. |
+| tableCaseSensitiveItemNames | `false`                                                      |    No     | table name case when `tableUseRealItemNames` is `true`. When set to `true`, item name case is preserved in table names and no suffix is used. When set to `false`, table names are lower cased and a numeric suffix is added. Please read [this](#case-sensitive-item-names) before enabling. |
+| tableIdDigitCount           | 4                                                            |    No     | when `tableUseRealItemNames` is `false` and thus table names are generated sequentially, this controls how many zero-padded digits are used in the table name.  With the default of 4, the first table name will end with `0001`. For migration from the MySQL persistence service, set this to 0. |
+| rebuildTableNames           | false                                                        |    No     | rename existing tables using `tableUseRealItemNames` and `tableIdDigitCount`. USE WITH CARE! Deactivate after Renaming is done! |
+| jdbc.maximumPoolSize        | configured per database in package `org.openhab.persistence.jdbc.db.*` |    No     | Some embedded databases can handle only one connection. See [this link](/~https://github.com/brettwooldridge/HikariCP/issues/256) for more information |
+| jdbc.minimumIdle            | see above                                                    |    No     | see above                                                    |
+| enableLogTime               | `false`                                                      |    No     | timekeeping                                                  |
 
 All item- and event-related configuration is done in the file `persistence/jdbc.persist`.
 
-To configure this service as the default persistence service for openHAB 2, add or change the line
+To configure this service as the default persistence service for openHAB, add or change the line
 
 ```
 org.openhab.core.persistence:default=jdbc
@@ -85,6 +86,13 @@ services/jdbc.cfg
 url=jdbc:postgresql://192.168.0.1:5432/testPostgresql
 ```
 
+### Case Sensitive Item Names
+
+To avoid numbered suffixes entirely, `tableUseRealItemNames` and `tableCaseSensitiveItemNames` must both be enabled.
+With this configuration, tables are named exactly like their corresponding items.
+In order for this to work correctly, the underlying operating system, database server and configuration must support case sensitive table names.
+For MySQL, see [MySQL: Identifier Case Sensitivity](https://dev.mysql.com/doc/refman/8.0/en/identifier-case-sensitivity.html) for more information.
+
 ### Migration from MySQL to JDBC Persistence Services
 
 The JDBC Persistence service can act as a replacement for the MySQL Persistence service.

diff --git a/...g.openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcBaseDAO.java b/...g.openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcBaseDAO.java
@@ -84,8 +84,9 @@ public class JdbcBaseDAO {
     protected String sqlIfTableExists = "SHOW TABLES LIKE '#searchTable#'";
     protected String sqlCreateNewEntryInItemsTable = "INSERT INTO #itemsManageTable# (ItemName) VALUES ('#itemname#')";
     protected String sqlCreateItemsTableIfNot = "CREATE TABLE IF NOT EXISTS #itemsManageTable# (ItemId INT NOT NULL AUTO_INCREMENT,#colname# #coltype# NOT NULL,PRIMARY KEY (ItemId))";
+    protected String sqlDropItemsTableIfExists = "DROP TABLE IF EXISTS #itemsManageTable#";
     protected String sqlDeleteItemsEntry = "DELETE FROM items WHERE ItemName=#itemname#";
-    protected String sqlGetItemIDTableNames = "SELECT itemid, itemname FROM #itemsManageTable#";
+    protected String sqlGetItemIDTableNames = "SELECT ItemId, ItemName FROM #itemsManageTable#";
     protected String sqlGetItemTables = "SELECT table_name FROM information_schema.tables WHERE table_type='BASE TABLE' AND table_schema='#jdbcUriDatabaseName#' AND NOT table_name='#itemsManageTable#'";
     protected String sqlCreateItemTable = "CREATE TABLE IF NOT EXISTS #tableName# (time #tablePrimaryKey# NOT NULL, value #dbType#, PRIMARY KEY(time))";
     protected String sqlInsertItemValue = "INSERT INTO #tableName# (TIME, VALUE) VALUES( #tablePrimaryValue#, ? ) ON DUPLICATE KEY UPDATE VALUE= ?";
@@ -266,7 +267,7 @@ public boolean doIfTableExists(ItemsVO vo) {
     public Long doCreateNewEntryInItemsTable(ItemsVO vo) {
         String sql = StringUtilsExt.replaceArrayMerge(sqlCreateNewEntryInItemsTable,
                 new String[] { "#itemsManageTable#", "#itemname#" },
-                new String[] { vo.getItemsManageTable(), vo.getItemname() });
+                new String[] { vo.getItemsManageTable(), vo.getItemName() });
         logger.debug("JDBC::doCreateNewEntryInItemsTable sql={}", sql);
         return Yank.insert(sql, null);
     }
@@ -280,9 +281,17 @@ public ItemsVO doCreateItemsTableIfNot(ItemsVO vo) {
         return vo;
     }
 
+    public ItemsVO doDropItemsTableIfExists(ItemsVO vo) {
+        String sql = StringUtilsExt.replaceArrayMerge(sqlDropItemsTableIfExists, new String[] { "#itemsManageTable#" },
+                new String[] { vo.getItemsManageTable() });
+        logger.debug("JDBC::doDropItemsTableIfExists sql={}", sql);
+        Yank.execute(sql, null);
+        return vo;
+    }
+
     public void doDeleteItemsEntry(ItemsVO vo) {
         String sql = StringUtilsExt.replaceArrayMerge(sqlDeleteItemsEntry, new String[] { "#itemname#" },
-                new String[] { vo.getItemname() });
+                new String[] { vo.getItemName() });
         logger.debug("JDBC::doDeleteItemsEntry sql={}", sql);
         Yank.execute(sql, null);
     }
@@ -306,8 +315,9 @@ public List<ItemsVO> doGetItemTables(ItemsVO vo) {
      * ITEM DAOs *
      *************/
     public void doUpdateItemTableNames(List<ItemVO> vol) {
-        if (!vol.isEmpty()) {
-            String sql = updateItemTableNamesProvider(vol);
+        logger.debug("JDBC::doUpdateItemTableNames vol.size = {}", vol.size());
+        for (ItemVO itemTable : vol) {
+            String sql = updateItemTableNamesProvider(itemTable);
             Yank.execute(sql, null);
         }
     }
@@ -416,13 +426,8 @@ protected String resolveTimeFilter(FilterCriteria filter, ZoneId timeZone) {
         return filterString;
     }
 
-    private String updateItemTableNamesProvider(List<ItemVO> namesList) {
-        logger.debug("JDBC::updateItemTableNamesProvider namesList.size = {}", namesList.size());
-        String queryString = "";
-        for (int i = 0; i < namesList.size(); i++) {
-            ItemVO it = namesList.get(i);
-            queryString += "ALTER TABLE " + it.getTableName() + " RENAME TO " + it.getNewTableName() + ";";
-        }
+    private String updateItemTableNamesProvider(ItemVO itemTable) {
+        String queryString = "ALTER TABLE " + itemTable.getTableName() + " RENAME TO " + itemTable.getNewTableName();
         logger.debug("JDBC::query queryString = {}", queryString);
         return queryString;
     }

diff --git a/....openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcDerbyDAO.java b/....openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcDerbyDAO.java
@@ -123,7 +123,7 @@ public boolean doIfTableExists(ItemsVO vo) {
     public Long doCreateNewEntryInItemsTable(ItemsVO vo) {
         String sql = StringUtilsExt.replaceArrayMerge(sqlCreateNewEntryInItemsTable,
                 new String[] { "#itemsManageTable#", "#itemname#" },
-                new String[] { vo.getItemsManageTable().toUpperCase(), vo.getItemname() });
+                new String[] { vo.getItemsManageTable().toUpperCase(), vo.getItemName() });
         logger.debug("JDBC::doCreateNewEntryInItemsTable sql={}", sql);
         return Yank.insert(sql, null);
     }

diff --git a/...openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcHsqldbDAO.java b/...openhab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcHsqldbDAO.java
@@ -101,7 +101,7 @@ public ItemsVO doCreateItemsTableIfNot(ItemsVO vo) {
     public Long doCreateNewEntryInItemsTable(ItemsVO vo) {
         String sql = StringUtilsExt.replaceArrayMerge(sqlCreateNewEntryInItemsTable,
                 new String[] { "#itemsManageTable#", "#itemname#" },
-                new String[] { vo.getItemsManageTable(), vo.getItemname() });
+                new String[] { vo.getItemsManageTable(), vo.getItemName() });
         logger.debug("JDBC::doCreateNewEntryInItemsTable sql={}", sql);
         return Yank.insert(sql, null);
     }

diff --git a/...hab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcPostgresqlDAO.java b/...hab.persistence.jdbc/src/main/java/org/openhab/persistence/jdbc/db/JdbcPostgresqlDAO.java
@@ -121,7 +121,7 @@ public ItemsVO doCreateItemsTableIfNot(ItemsVO vo) {
     public Long doCreateNewEntryInItemsTable(ItemsVO vo) {
         String sql = StringUtilsExt.replaceArrayMerge(sqlCreateNewEntryInItemsTable,
                 new String[] { "#itemsManageTable#", "#itemname#" },
-                new String[] { vo.getItemsManageTable(), vo.getItemname() });
+                new String[] { vo.getItemsManageTable(), vo.getItemName() });
         logger.debug("JDBC::doCreateNewEntryInItemsTable sql={}", sql);
         return Yank.insert(sql, null);
     }