NandiniN
Databricks Employee
Databricks Employee

You can choose instance types based on the workload:

  • Memory Optimized instances for machine learning workloads, tasks involving a lot of shuffle and spills, and when Spark caching is needed.
  • Compute Optimized instances for structured streaming jobs, ELT with full scan and no data reuse, and to run OPTIMIZE and Z-order Delta commands.
  • Storage Optimized instances to leverage Delta caching, for ML and DL workloads with data caching, and for ad hoc and interactive data analysis.
  • General Purpose instances for jobs without specific requirements and to run VACUUM Delta command.