Hi all, how's everyone doing? Topic: Databricks Metastore Cluster Details: Databricks runtime 10.4 ML LTS Apache Spark 3.2.1, Scala 2.12 Problem description: We have an issue with our metastore slowing down read and write to our delta tables. We initially attempted to communicate with databricks support to understand the Metastore is Down Issue. We were told it's due to BoneCP trying to established failed connections from the connection pool etc. Attempt 1: The resolution was to try using HikariCP instead of BoneCP, after adding HikariCP datanucleas jar files to /databricks/hive.

spark.hadoop.datanucleus.connectionPoolingType hikari

We got the same error as below. Attempt 2: Docs : • https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/10.4ml • https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/10.4#hikaricp-is-now-the-default-hive-metastore-connection-pool installed packages: x So 10.4 ML is different to 10.4 in that 10.4 ML is not using hikari (and has the below libraries missing from the databricks Libraries UI) • We installed the following (because this is the standard from 10.4 non-ml version):

org.datanucleus:datanucleus-api-jdo:4.2.4
org.datanucleus:datanucleus-core:4.1.17
org.datanucleus:datanucleus-rdbms:4.1.19
org.datanucleus:javax.jdo:3.2.0-m3

configuration that we've tried:

spark.hadoop.datanucleus.connectionPoolingType hikari

spark.hadoop.datanucleus.connectionPoolingType HikariCP

spark.databricks.hive.metastore.client.pool.type hikari

spark.databricks.hive.metastore.client.pool.type HikariCP

All jar files were added and we are getting the following error after restarting. ``org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)`

... 128 more

`Caused by: java.lang.Throwable: Attempt to invoke the "hikari" plugin to create a ConnectionPool gave an error : The connection pool plugin of type "hikari" was not found in the CLASSPATH!`` Any one with knowledge on how to resolve this issue, we would appreciate your help. Thanks

Please contact your Databricks rep. This channel is for the open-source Delta Lake project

Thanks for the reply, I will do so :-)