Unable to run 2 different applications with the same class name on a cluster

brickster_2018
Databricks Employee
Databricks Employee

‎06-23-2021 11:40 PM - last edited on ‎03-20-2025 10:11 AM by Community Manager

I have two jars with the same class name. It works fine on yarn. When trying to run these jars on the Databricks cluster, I run into issues. Why Databricks is having this limitation?

Labels:
0 Kudos

brickster_2018
Databricks Employee
Databricks Employee

‎06-23-2021 11:52 PM

When you run the jobs in Yarn, those are 2 different applications getting submitted on Yarn. Hence each application will have a separate Spark driver JVM's.

In Databricks, a cluster has one JVM for the Spark driver. When applications with the same name are submitted on the same JVM, it's possible the classes are loaded from the incorrect jars.

Mitigations/Solution:

  • Use an on-demand cluster for your jobs. This will ensure one jar uses a dedicated cluster.
  • Change the class name in one of the classes to avoid conflict.

View solution in original post

0 Kudos