Comment
AG2
Databricks Partner
‎11-27-2023 02:07 AM
‎11-27-2023 02:07 AM

Some more Performance optimization techniques one should follow 

  1. Use the latest LTS version of Databricks Runtime, The latest Databricks runtime is almost always faster than the one before it(DBR 14.0 is GA with Spark 3.5.0, UDAF for Python, Row-level concurrency, Spark Connect)
  2. Use Photon - fastest Spark execution
  3. Restart long-running clusters periodically
  4. Use Cluster Policies to enforce best practices!