photon is being used by a job or not
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
We have lots of customers using many job as well as interactive clusters with photon enabled which is drastically increasing the cost .
We would like to know if there is any table in system or any details that we can get through API that lists if the jobs running are actually using photon or it is getting ignored for this job which is costing us a lot for no use
Please let me know if there is a common point to check which jobs are actually enabled with photon and are not using it in real time run
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi @GSRuh,
If your cluster has enabled photon you will be charged for it regardless if it benefits your workload or not.
You can find which clusters using photon by checking system table `usage`, searching for the records where `sku_name` contains 'PHOTON' or where `product_features.is_photon` equals `true`. You can use query below to find all your clusters which are using Photon.
select distinct c.account_id, c.workspace_id, c.cluster_id, c.cluster_name, u.sku_name, c.create_time
from system.billing.usage u
join system.compute.clusters c
on u.account_id = c.account_id
and u.workspace_id = c.workspace_id
and u.usage_metadata.cluster_id = c.cluster_id
where u.sku_name like '%PHOTON%'
or u.product_features.is_photon = true;
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Thanks @rasskazovp,
I can see the list of clusters enabled with Photon in the usage table. However, it would be really helpful if we could determine whether Photon is actually being used by a job during its run. This information would allow us to save costs and utilize resources more effectively.
It would be beneficial to have a place where we can see if Photon is being used by a job or if it is not supported. Additionally, having a list of all jobs where Photon is enabled but not supported would be very useful.
please let us know if we can achieve this through API
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hey @sruthianki ,
I think you can’t check via API if Photon was used, but you can review the Spark UI description and check Photon logs to confirm in which jobs it was utilized. However, I can suggest the following recommendations to help you decide whether enabling Photon is worthwhile
Use Photon if:
•You work with Delta Lake and large volumes of data.
•You run SQL-intensive queries (aggregations, joins, etc.).
•You want to optimize costs in Databricks clusters.
Photon is not ideal if:
•You execute a lot of pure Python code, as Photon does not accelerate operations outside of SQL/DataFrames.
•You use ML or AI in Spark MLlib, because Photon does not optimize these processes.
🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi @sruthianki ,
If you want to check if the job is really using photon or not you can check the SQL query plan in spark UI for its stages and the metrics will highlighted in yellow colour.

