cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Hubert-Dudek
by • Esteemed Contributor III
  • 2696 Views
  • 2 replies
  • 9 kudos

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function...

databricks Photon is a next-generation engine on the Databricks Lakehouse Platform that provides speedy query performance at a low cost.- Its function coverage is growing, and UDF under Photon is coming, which can bring significant improvements in us...

ezgif-5-724cb0ccf8
Data Engineering
Reply
  • 2696 Views
  • 2 replies
  • 9 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 9 kudos

 

  • 9 kudos
1 More Replies
kk007
by • New Contributor III
  • 3012 Views
  • 4 replies
  • 4 kudos

Photon engine throws error "JSON document exceeded maximum allowed size 400.0 MiB"

I am reading a 83MB json file using " spark.read.json(storage_path)", when I display the data is seems displaying fine, but when I try command line count, it complains about file size , being more than 400MB, which is not true.Photon JSON reader erro...

Data Engineering
Reply
  • 3012 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Kamal Kumar​ :The error message suggests that the JSON document size is exceeding the maximum allowed size of 400MB. This could be caused by one or more documents in your JSON file being larger than this limit. It is not a bug, but a limitation set ...

  • 4 kudos
3 More Replies
HariSelvarajan
by • Databricks Employee
  • 726 Views
  • 0 replies
  • 5 kudos

DAIWT22_RadicalSpeenInLakehouse_Photon

Topic: Radical Speed on the Lakehouse: Photon under the hoodI am Hari and I works as a Specialist Solutions Architect at Databricks. I specialise in Data engineering and Cloud platforms problems helping client in EMEA.Purpose:I recently presented a t...

Data Engineering
Reply
  • 726 Views
  • 0 replies
  • 5 kudos
Pragan
by • New Contributor
  • 3311 Views
  • 3 replies
  • 1 kudos

Resolved! Cluster doesn't support Photon with Docker Image enabled

I enabled Photon 9.1 LTS DBR in cluster that was already using Docker Image of the latest version, when I ran a SQL QUery using my cluster, I could not see any Photon engine working in my executor that should be actually running in Photon Engine.When...

Data Engineering
Reply
  • 3311 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello @Praganessh S​ , Photon is currently in Public Preview. The only way to use it is to explicitly run Databricks-provide Runtime images which contain it. Please see: https://docs.databricks.com/runtime/photon.html#databricks-clustersandhttps://do...

  • 1 kudos
2 More Replies
MoJaMa
by • Databricks Employee
  • 976 Views
  • 1 replies
  • 0 kudos

What is this Photon Engine I keep hearing about?

Data Engineering
Reply
  • 976 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
MoJaMa
Databricks Employee
  • 0 kudos

It's our new high-performance runtime, using a native vectorized engine developed in C++.Please see our blog for a great overview. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databri...

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 1368 Views
  • 1 replies
  • 0 kudos

How is the photon engine different to catalyst optimizer

How is the photon engine different to catalyst optimizer

Data Engineering
Reply
  • 1368 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

I got this question from some customers and I want ti clarify here tooI think we are conflating two things:Catalyst optimizer is about coming up "Steps to take to execute the query". For example, the optimizer will decide how and when to do the join...

  • 0 kudos
Labels