photon is being used by a job or not

[sruthianki]

We have lots of customers using many job as well as interactive clusters with photon enabled which is drastically increasing the cost .

We would like to know if there is any table in system or any details that we can get through API that lists if the jobs running are actually using photon or it is getting ignored for this job which is costing us a lot for no use

Please let me know if there is a common point to check which jobs are actually enabled with photon and are not using it in real time run

[rasskazovp]

Hi @GSRuh,

If your cluster has enabled photon you will be charged for it regardless if it benefits your workload or not.

You can find which clusters using photon by checking system table `usage`, searching for the records where `sku_name` contains 'PHOTON' or where `product_features.is_photon` equals `true`. You can use query below to find all your clusters which are using Photon.

select distinct c.account_id, c.workspace_id, c.cluster_id, c.cluster_name, u.sku_name, c.create_time
from system.billing.usage u
join system.compute.clusters c
  on u.account_id = c.account_id
  and u.workspace_id = c.workspace_id
  and u.usage_metadata.cluster_id = c.cluster_id
where u.sku_name like '%PHOTON%'
or u.product_features.is_photon = true;

You can find here how enable system tables for Azure, AWS

[sruthianki]

Thanks @rasskazovp,

I can see the list of clusters enabled with Photon in the usage table. However, it would be really helpful if we could determine whether Photon is actually being used by a job during its run. This information would allow us to save costs and utilize resources more effectively.

It would be beneficial to have a place where we can see if Photon is being used by a job or if it is not supported. Additionally, having a list of all jobs where Photon is enabled but not supported would be very useful.
 
please let us know if we can achieve this through API 

[Isi]

Hey @sruthianki ,

I think you can’t check via API if Photon was used, but you can review the Spark UI description and check Photon logs to confirm in which jobs it was utilized. However, I can suggest the following recommendations to help you decide whether enabling Photon is worthwhile

Use Photon if:

•You work with Delta Lake and large volumes of data.
•You run SQL-intensive queries (aggregations, joins, etc.).
•You want to optimize costs in Databricks clusters.

Photon is not ideal if:

•You execute a lot of pure Python code, as Photon does not accelerate operations outside of SQL/DataFrames.
•You use ML or AI in Spark MLlib, because Photon does not optimize these processes.

🙂

[Sidhant07]

Hi @sruthianki ,

If you want to check if the job is really using photon or not you can check the SQL query plan in spark UI for its stages and the metrics will highlighted in yellow colour.