cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User16783854657
by • New Contributor III
  • 1850 Views
  • 4 replies
  • 6 kudos

How do I know how much of a query/job used Photon?

I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?

Data Engineering
Reply
  • 1850 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
venkat09
New Contributor III
  • 6 kudos

Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.

  • 6 kudos
3 More Replies
aladda
by • Honored Contributor II
  • 2987 Views
  • 1 replies
  • 0 kudos

Resolved! How many columns does Delta Engine collect stats on for a Delta Table

Data Engineering
Reply
  • 2987 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

by default a delta table has stats collected on the first 32 columns. This setting can be configured using the following.set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3However there's a time trade-off to having a large n...

  • 0 kudos
User16826994223
by • Honored Contributor III
  • 242 Views
  • 0 replies
  • 0 kudos

databricks.com

Can I create a Delta Lake table on Databricks and query it with open-source Spark?Yes, in order to do this, you would install Open Source Spark and Delta Lake, both are open source. Delta Engine, which is only available on Databricks, will make delta...

Data Engineering
Reply
  • 242 Views
  • 0 replies
  • 0 kudos
Labels