@Alejandro Martinez​ :

Hi! Your template seems to be a good starting point for configuring a SparkSession in Databricks. However, there are some new recommendations that you can consider for Databricks runtime versions v10-11-12. Here are some suggestions:

1. Use Databricks Delta 1.0.0 or higher - Databricks Delta has been updated and improved since its preview release. It is now recommended to use Delta version 1.0.0 or higher, which includes many stability and performance improvements.
2. Configure Spark shuffle partitions - Configuring the number of partitions in Spark shuffle can significantly improve the performance of your Spark jobs. You can set the number of shuffle partitions based on the size of your data and the size of your cluster.
3. Enable automatic coalescing of small files - Enabling automatic coalescing of small files can help reduce the number of small files in your Delta tables, which can improve query performance.
4. Use the optimal file format - Different file formats have different performance characteristics. For example, Delta tables are optimized for performance and reliability, whereas Parquet is optimized for storage efficiency. Consider using the file format that best meets your needs.
5. Use adaptive query execution - Adaptive query execution is a feature that can automatically adjust the execution plan of a query based on the characteristics of the data and the cluster. It can improve the performance of your Spark jobs in many cases.

Here is an updated template that includes these recommendations:

spark_session = SparkSession.builder \
    .config("spark.databricks.delta.retentionDurationCheck.enabled", "false") \
    .config("spark.sql.shuffle.partitions", "500") \
    .config("spark.databricks.delta.optimizeWrite.enabled", "true") \
    .config("spark.databricks.delta.autoCompact.enabled", "true") \
    .config("spark.databricks.delta.join.preferBroadcastHashJoin", "true") \
    .config("spark.sql.adaptive.enabled", "true") \
    .getOrCreate()

Please note that these recommendations may not be suitable for all use cases, so you should evaluate them based on your specific requirements and workload characteristics.