databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.
- Its function coverage is growing, and UDF under Photon is coming, which can bring significant improvements in using UDFs.
- To enable Photon acceleration, select the Use Photon Acceleration checkbox when you create the cluster.
- Photon supports SQL and equivalent DataFrame operations against Delta and Parquet tables and accelerates queries that process a significant amount of data (100GB+) and include aggregations and joins.
- Photon provides faster performance when data is accessed repeatedly from the disk cache.
- Photon offers robust scan performance on tables with many columns and small files.
