Databricks Cluster is going down after installing the external library

[DineshKumar]

I have created a Databricks cluster with below configurations.

Databricks Runtime Version
13.2 ML (includes Apache Spark 3.4.0, Scala 2.12)

Node type
i3.xlarge
30.5 GB Memory, 4 Cores

I created a notebook and trying to load the Mysql table which resides in AWS RDS.

So, I have installed this jar file `mysql-connector-j-8.0.33.jar` into my cluster then I tried executing the below code in a notebook.

table = "test_table"
database_host = "test-db-dev.cluster-ro-pkh5s13hvg.us-west-2.rds.amazonaws.com"
database_port = "3306" 
database_name = "test_db"
user = "test_user"
password = "12345678"
url = f"jdbc:mysql://{database_host}:{database_port}/{database_name}"

df_rds_table = spark.read \
    .format("jdbc") \
    .option("driver","com.mysql.cj.jdbc.Driver") \
    .option("url", url) \
    .option("dbtable", table) \
    .option("user", user) \
    .option("password", password) \
    .load()

The command execution was running for long time and finally ended up with failed status by throwing the below error.

java.lang.Exception: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$2(HiveExternalCatalog.scala:167)
	at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:114)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:152)
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:364)
	at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:151)
	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:323)
	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:302)
	at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:297)
	at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:60)
	at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$resourceLoader$1(HiveSessionStateBuilder.scala:67)
	at org.apache.spark.sql.hive.HiveSessionResourceLoader.client$lzycompute(HiveSessionStateBuilder.scala:166)
	at org.apache.spark.sql.hive.HiveSessionResourceLoader.client(HiveSessionStateBuilder.scala:166)
	at org.apache.spark.sql.hive.HiveSessionResourceLoader.$anonfun$addJar$1(HiveSessionStateBuilder.scala:170)
	at org.apache.spark.sql.hive.HiveSessionResourceLoader.$anonfun$addJar$1$adapted(HiveSessionStateBuilder.scala:169)
	at scala.collection.immutable.List.foreach(List.scala:431)
.
.
.

I observed that after installing the library I could see the below status in Cluster's Event log.

Event Type                          Time                                         Message

METASTORE_DOWN     2023-07-27 18:22:11 CDT      Metastore is down.

Kindly assist me on this.

[daniel_sahal]

@DineshKumar 
I'm not 100% sure what the issue could be, but my lucky guess is that you're reinstalling the JDBC driver that is used to connect to Hive Metastore (it's on MySQL though).

Try to use the one that is already installed: https://docs.databricks.com/external-data/mysql.html

[DineshKumar]

@daniel_sahal,

I tried uninstalling the JDBC driver but this time I am getting the below classNotFound error.

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

 Also, I tried using the one (jdbc) from above link which you shared. I am getting the below error.

java.sql.SQLInvalidAuthorizationSpecException: Could not connect to address=(host=test-db-dev.cluster-ro-pkh5s13hvg.us-west-2.rds.amazonaws.com)
(port=3306)(type=master) : (conn=12345676) Access denied for user 'test_user'@'10.2255.1134.111' (using password: YES)

[daniel_sahal]

@DineshKumar 
Don't try to remove it. Just don't install it 🙂

About the error - it's clearly permissions issue to the database. Firstly check if the user has got an access to the database. Secondly, check if there's no firewall - if yes make sure that the address range that you're connecting from is whitelisted.

[DineshKumar]

@daniel_sahal 

I have checked that the user has read only access to the database. I connected through Dbeaver with same credentials. 

Secondly, can you guide me on how to check if there is no firewall or not. Where can I check this. Kindly guide me on this.

Thanks!

[daniel_sahal]

@DineshKumar 
Please check this stackoverflow question:
https://stackoverflow.com/questions/8380797/enable-remote-mysql-connection-error-1045-28000-access-d...

[Debayan]

Hi, The below error describes that there is an issue connecting to the host from Databricks, you can find more details about the network configurations here at https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html

Could not connect to address=(host=test-db-dev.cluster-ro-pkh5s13hvg.us-west-2.rds.amazonaws.com)
(port=3306)(type=master) : (conn=12345676) Access denied for user 'test_user'@'10.2255.1134.111' (using password: YES)