cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Few things you should not do in Databricks!

brickster_2018
Esteemed Contributor

Few things you should not do in Databricks!

1 REPLY 1

brickster_2018
Esteemed Contributor

Compared to OSS Spark, these are few things the users don't have to worry about when running the same job on Databricks.

  • Memory management: Databricks use an internal formula to allocate the Driver and executor heap based on the size of the instance. So it's safe to remove the configurations like spark.driver.memory, spark.executor.memory.
  • The number of executors: Number of executors, tasks per executors, the total number of executors-related configurations are not necessary when running jobs in Datatbicks. By default we launch one executor per instance. And the number of tasks that can run on the executor will be equal to the total number of cores on the instance.
  • Avoid using spark.stop, sc.stop : Databricks manages the cluster shutdown on its own, so stopping the context or spark session is not required.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group