cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to run 2 different applications with the same class name on a cluster

brickster_2018
Databricks Employee
Databricks Employee

I have two jars with the same class name. It works fine on yarn. When trying to run these jars on the Databricks cluster, I run into issues. Why Databricks is having this limitation?

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Databricks Employee
Databricks Employee

When you run the jobs in Yarn, those are 2 different applications getting submitted on Yarn. Hence each application will have a separate Spark driver JVM's.

In Databricks, a cluster has one JVM for the Spark driver. When applications with the same name are submitted on the same JVM, it's possible the classes are loaded from the incorrect jars.

Mitigations/Solution:

  • Use an on-demand cluster for your jobs. This will ensure one jar uses a dedicated cluster.
  • Change the class name in one of the classes to avoid conflict.

View solution in original post

1 REPLY 1

brickster_2018
Databricks Employee
Databricks Employee

When you run the jobs in Yarn, those are 2 different applications getting submitted on Yarn. Hence each application will have a separate Spark driver JVM's.

In Databricks, a cluster has one JVM for the Spark driver. When applications with the same name are submitted on the same JVM, it's possible the classes are loaded from the incorrect jars.

Mitigations/Solution:

  • Use an on-demand cluster for your jobs. This will ensure one jar uses a dedicated cluster.
  • Change the class name in one of the classes to avoid conflict.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group