Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)?

Sainath368 — Mon, 01 Sep 2025 11:08:47 GMT

Hi everyone,

We’re currently reviewing the performance impact of enabling Photon acceleration on our Databricks jobs, particularly those involving table maintenance tasks. Our job includes three main operations: OPTIMIZE, VACUUM, and ANALYZE_COMPUTE_STATS. We’ve observed that enabling Photon significantly improves the performance of the ANALYZE_COMPUTE_STATS task—it runs much faster when Photon is enabled on the cluster.

Given that, I’m wondering if enabling Photon for the other two tasks (OPTIMIZE and VACUUM) would also lead to better performance or reduced job time. Has anyone experienced improvements in these tasks with Photon?

Also, more generally, I’d like to understand which types of tasks or workloads benefit most from Photon acceleration.

Any insights, benchmarks, or shared experiences would be really helpful. Thanks!

Re: Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STAT

szymon_dybczak — Mon, 01 Sep 2025 11:48:28 GMT

Hi @Sainath368 ,

I wouldn't use photon for this kind of task. You should use it primarly for ETL transformations where it shines.
VACUUM and OPTIMIZE are more of maintenance tasks and using photon would be pricey overkill here.

According to documentation, it is recommended to enable Photon for workloads with the following characteristics:

ETL pipelines consisting of Delta MERGE operations
Writing large volumes of data to cloud storage (Delta/Parquet)
Scans of large data sets, joins, aggregations and decimal computations
Auto Loader to incrementally and efficiently process new data arriving in storage
Interactive/ad hoc queries using SQL

Regarding advantages of Photon:

Accelerated queries that process a significant amount of data (> 100GB) and include aggregations and joins
Faster performance when data is accessed repeatedly from the Delta cache
More robust scan/read performance on tables with many columns and many small files
Faster Delta writing using UPDATE, DELETE, MERGE INTO, INSERT, and CREATE TABLE AS SELECT
Join improvements

Comprehensive Guide to Optimize Data Workloads | Databricks

For instance, for VACUUM databricks recommends to use compute optimized instances. And since OPTIMIZE is also compute intensive I guess it also applies to it.

topic Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)? in Data Engineering

Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)?

Re: Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STAT