Allie Ubisse

02/20/2023, 3:18 PM
Hi all, how's everyone doing? We have an issue with our metastore slowing down read and write to our delta tables. Topic: Databricks Metastore Cluster Details: Databricks runtime 10.4 ML LTS Apache Spark 3.2.1, Scala 2.12 We have initially attempted to communicate with databricks support to understand the Metastore is Down Issue. We were told it's due to BoneCP trying to established failed connections from the connection pool etc. Attempt 1: The resolution was to try using HikariCP instead of BoneCP, after adding HikariCP datanucleas jar files to /databricks/hive.
spark.hadoop.datanucleus.connectionPoolingType hikari
We got the same error as below. Attempt 2: Docs : • installed packages: x So 10.4 ML is different to 10.4 in that 10.4 ML is not using hikari (and has the below libraries missing from the databricks Libraries UI) • We installed the following (because this is the standard from 10.4 non-ml version):
Copy code
configuration that we've tried:
spark.hadoop.datanucleus.connectionPoolingType hikari
spark.hadoop.datanucleus.connectionPoolingType HikariCP
spark.databricks.hive.metastore.client.pool.type hikari
spark.databricks.hive.metastore.client.pool.type HikariCP
All jar files were added and we are getting the following error after restarting. ``org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(`
... 128 more
`Caused by: java.lang.Throwable: Attempt to invoke the "hikari" plugin to create a ConnectionPool gave an error : The connection pool plugin of type "hikari" was not found in the CLASSPATH!`` Any one with knowledge on how to resolve this issue, we would appreciate your help. Thanks