cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

photon is being used by a job or not

sruthianki
New Contributor II

We have lots of customers using many job as well as interactive clusters with photon enabled which is drastically increasing the cost .

We would like to know if there is any table in system or any details that we can get through API that lists if the jobs running are actually using photon or it is getting ignored for this job which is costing us a lot for no use

Please let me know if there is a common point to check which jobs are actually enabled with photon and are not using it in real time run

4 REPLIES 4

rasskazovp
New Contributor II

Hi @GSRuh,

If your cluster has enabled photon you will be charged for it regardless if it benefits your workload or not.

You can find which clusters using photon by checking system table `usage`, searching for the records where `sku_name` contains 'PHOTON' or where `product_features.is_photon` equals `true`. You can use query below to find all your clusters which are using Photon.

select distinct c.account_id, c.workspace_id, c.cluster_id, c.cluster_name, u.sku_name, c.create_time
from system.billing.usage u
join system.compute.clusters c
  on u.account_id = c.account_id
  and u.workspace_id = c.workspace_id
  and u.usage_metadata.cluster_id = c.cluster_id
where u.sku_name like '%PHOTON%'
or u.product_features.is_photon = true;

You can find here how enable system tables for Azure, AWS

sruthianki
New Contributor II

Thanks @rasskazovp,

I can see the list of clusters enabled with Photon in the usage table. However, it would be really helpful if we could determine whether Photon is actually being used by a job during its run. This information would allow us to save costs and utilize resources more effectively.

It would be beneficial to have a place where we can see if Photon is being used by a job or if it is not supported. Additionally, having a list of all jobs where Photon is enabled but not supported would be very useful.
 
please let us know if we can achieve this through API 

 

Isi
New Contributor III

Hey @sruthianki ,

I think you can’t check via API if Photon was used, but you can review the Spark UI description and check Photon logs to confirm in which jobs it was utilized. However, I can suggest the following recommendations to help you decide whether enabling Photon is worthwhile

Use Photon if:

•You work with Delta Lake and large volumes of data.
•You run SQL-intensive queries (aggregations, joins, etc.).
•You want to optimize costs in Databricks clusters.


Photon is not ideal if:

•You execute a lot of pure Python code, as Photon does not accelerate operations outside of SQL/DataFrames.
•You use ML or AI in Spark MLlib, because Photon does not optimize these processes.

🙂

Sidhant07
Databricks Employee
Databricks Employee

Hi @sruthianki ,

If you want to check if the job is really using photon or not you can check the SQL query plan in spark UI for its stages and the metrics will highlighted in yellow colour.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group