Advantage of using Photon Engine
The following summarizes the advantages of Photon:
- Supports SQL and equivalent DataFrame operations against Delta and Parquet tables.
- Expected to accelerate queries that process a significant amount of data (100GB+) and include aggregations and joins.
- Data is accessed repeatedly and likely in the Delta Lake cache.
- More robust scan performance on tables with many columns and many small files.
- Faster Delta and Parquet writing using UPDATE, DELETE, MERGE INTO, INSERT and CREATE TABLE AS SELECT especially for wide tables (hundreds to thousands of columns).
- Photon replaces sort-merge joins with hash-joins.
More details are on - https://docs.databricks.com/runtime/photon.html
