- 3547 Views
- 4 replies
- 6 kudos
How do I know how much of a query/job used Photon?
I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?
- 3547 Views
- 4 replies
- 6 kudos
- 6 kudos
Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.
- 6 kudos
- 4951 Views
- 1 replies
- 0 kudos
- 4951 Views
- 1 replies
- 0 kudos
- 0 kudos
by default a delta table has stats collected on the first 32 columns. This setting can be configured using the following.set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3However there's a time trade-off to having a large n...
- 0 kudos
- 476 Views
- 0 replies
- 0 kudos
databricks.com
Can I create a Delta Lake table on Databricks and query it with open-source Spark?Yes, in order to do this, you would install Open Source Spark and Delta Lake, both are open source. Delta Engine, which is only available on Databricks, will make delta...
- 476 Views
- 0 replies
- 0 kudos