cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Hubert-Dudek
by • Esteemed Contributor III
  • 2399 Views
  • 2 replies
  • 9 kudos

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function...

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function coverage is growing, and UDF under Photon is coming, which can bring significant improvements in us...

ezgif-5-724cb0ccf8
Data Engineering
Reply
  • 2399 Views
  • 2 replies
  • 9 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 9 kudos

 

  • 9 kudos
1 More Replies
JLSy
by • New Contributor III
  • 11971 Views
  • 5 replies
  • 6 kudos

cannot convert Parquet type INT64 to Photon type string

I am receiving an error similar to the post in this link: https://community.databricks.com/s/question/0D58Y00009d8h4tSAA/cannot-convert-parquet-type-int64-to-photon-type-doubleHowever, instead of type double the error message states that the type can...

Data Engineering
Reply
  • 11971 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

@John Laurence Sy​ :It sounds like you are encountering a schema conversion error when trying to read in a Parquet file that contains an INT64 column that cannot be converted to a string type. This error can occur when the Parquet file has a schema t...

  • 6 kudos
4 More Replies
User16783854657
by • New Contributor III
  • 3136 Views
  • 4 replies
  • 6 kudos

How do I know how much of a query/job used Photon?

I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?

Data Engineering
Reply
  • 3136 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
venkat09
New Contributor III
  • 6 kudos

Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.

  • 6 kudos
3 More Replies
auser85
by • New Contributor III
  • 3846 Views
  • 2 replies
  • 2 kudos

cannot convert Parquet type INT64 to Photon type double

I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`These are my option...

Data Engineering
Reply
  • 3846 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

hey @Andrew Fogarty​ I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsS...

  • 2 kudos
1 More Replies
lawrence009
by • Contributor
  • 2662 Views
  • 4 replies
  • 8 kudos

Photon does not fully support the query because of dynamic pruning

Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?This is the code I ran:CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ...

Data Engineering
Reply
  • 2662 Views
  • 4 replies
  • 8 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Lawrence Chen​, Please tell us your DBR version.

  • 8 kudos
3 More Replies
Sajid1
by • Contributor
  • 3481 Views
  • 5 replies
  • 6 kudos

Resolved! Photon Acceleration not getting enabled for ML runtime in Azure

I tried to enable the photon acceleration in ML runtime 9.1 LTS ML (Scala 2.12,Spark 3.1.2) but getting error "selected runtime version does not support photon".I tried for other versions of ML runtime with single and multinode , access mode being s...

Data Engineering
Reply
  • 3481 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Sajid Thavalengal Rahiman​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 6 kudos
4 More Replies
AJMorgan591
by • New Contributor II
  • 2727 Views
  • 4 replies
  • 0 kudos

Temporarily disable Photon

Is it possible to temporarily disable Photon?I have a large workload that greatly benefits from Photon apart from a specific operation therein that is actually slowed by Photon. It's not worth creating a separate cluster for this operation however, s...

Data Engineering
Reply
  • 2727 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Aaron Morgan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
3 More Replies
User16826994223
by • Honored Contributor III
  • 1154 Views
  • 1 replies
  • 0 kudos

Some of the limitation I see In docs of photon until now july 2021 is  Works on Delta and Parquet tables only for both read and write.Does not suppor...

Some of the limitation I see In docs of photon until now july 2021 is Works on Delta and Parquet tables only for both read and write.Does not support the following data types:MapArrayDoes not support window and sort operatorsDoes not support Spark S...

Data Engineering
Reply
  • 1154 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Excellent post, Thank you @Kunal Gaurav​ !

  • 0 kudos
User16826987838
by • Contributor
  • 1130 Views
  • 1 replies
  • 0 kudos

What type of aws instance and how many are used for an L sized Databricks SQL(SQLA) cluster ?

What type of aws instance and how many are used for an L sized Databricks SQL(SQLA) cluster with Photon enabled

Data Engineering
Reply
  • 1130 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Taha
Contributor
  • 0 kudos

L size is 16 workers of i3.8xlarge

  • 0 kudos
Srikanth_Gupta_
by • Valued Contributor
  • 788 Views
  • 1 replies
  • 1 kudos

Can we use Photon for batch and streaming process instead of Spark, when will be available for public?

Data Engineering
Reply
  • 788 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
aladda
Honored Contributor II
  • 1 kudos

Photon is supported for batch workloads today and is the standard on Databricks SQL clusters and available as an option for Automated and Interactive clusters. And photon is in public preview today so available as an option for everyone. See this lin...

  • 1 kudos
Anonymous
by • Not applicable
  • 782 Views
  • 1 replies
  • 0 kudos

Photon usage

How do I know how much of a query/job used Photon?

Data Engineering
Reply
  • 782 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

If you are using Photon on Databricks SQLClick the Query History icon on the sidebar.Click the line containing the query you’d like to analyze.On the Query Details pop-up, click Execution Details.Look at the Task Time in Photon metric at the bottom.

  • 0 kudos
User16826992666
by • Valued Contributor
  • 1141 Views
  • 1 replies
  • 0 kudos

Resolved! In Databricks SQL how can I tell if my query is using Photon?

I have turned Photon on in my endpoint, but I don't know if it's actually being used in my queries. Is there some way I can see this other than manually testing queries with Photon turned on and off?

Data Engineering
Reply
  • 1141 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Digan_Parikh
Valued Contributor
  • 0 kudos

@Trevor Bishop​ If you go to the History tab in DBSQL, click on the specific query and look at the execution details. At the bottom, you will see "Task time in Photon".

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 1209 Views
  • 1 replies
  • 0 kudos

What is Photon in DataBricks

Hey I am new to Databricks and heard of photon , which is the fastest engine developed by Databricks , Will it make the query faster , what about Concurrency of the queries , will it increase

Data Engineering
Reply
  • 1209 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Mooune_DBU
Valued Contributor
  • 0 kudos

Photon is databrick's brand new native vectorized engine developed in C++ for improved query performance (speed and concurrency). It integrates directly with the Databricks Runtime and Spark, meaning no code changes are required to use Photon. At thi...

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 785 Views
  • 1 replies
  • 0 kudos

Start photon cluster

How to start a photon cluster, where I can fins the pricing of photon Cluster

Data Engineering
Reply
  • 785 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
craig_ng
New Contributor III
  • 0 kudos

As of the time of this message, Photon availability in the Data Science & Engineering workspace in Public Preview on AWS. You can reference our docs for instructions on how to provision a cluster using a Photon-enabled runtime. As for pricing, we tre...

  • 0 kudos
MoJaMa
by • Valued Contributor II
  • 905 Views
  • 1 replies
  • 0 kudos

What is this Photon Engine I keep hearing about?

Data Engineering
Reply
  • 905 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Valued Contributor II
  • 0 kudos

It's our new high-performance runtime, using a native vectorized engine developed in C++.Please see our blog for a great overview. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databri...

  • 0 kudos
Labels