cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
tlecomte
by • New Contributor III
  • 4977 Views
  • 6 replies
  • 3 kudos

Resolved! Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch

Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...

Data Engineering
Reply
  • 4977 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
Lingesh
Databricks Employee
  • 3 kudos

It's not recommended to have AQE on a Streaming query for the same reason you shared in the description. It has been documented here

  • 3 kudos
5 More Replies
brickster_2018
by • Databricks Employee
  • 1208 Views
  • 1 replies
  • 0 kudos

What are the best practices for Adaptive query execution

What are common configurations used and which workload will get benefit

Data Engineering
Reply
  • 1208 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
amr
Databricks Employee
  • 0 kudos

Leave it turned on. the bet is with each Spark version released AQE will get better and better and eventually will lead to a much more performance optimisation plan than manually trying to tune it.

  • 0 kudos
User16826992666
by • Valued Contributor
  • 1148 Views
  • 1 replies
  • 0 kudos

Resolved! Do I still need to use skew join hints if I have Adaptive Query Execution enabled?

From what I have read about AQE it seems to do a lot of what skew join hints did automatically. So should I still be using skew hints in my queries? Is there harm in using them?

Data Engineering
Reply
  • 1148 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

With AQE Databricks  has the most up-to-date accurate statistics at the end of a query stage and can opt for a better physical strategy and or do optimizations that used to require hints,In the case of skew join hints, is recommended to rely on AQE...

  • 0 kudos
User16783855117
by • Contributor II
  • 832 Views
  • 0 replies
  • 0 kudos

Is there a way to know if Adaptive Query Execution with Spark 3 has changed my Spark plan?

From the demo notebook located here (https://databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html) it seems like the approach to demonstrate AQE was working was to first calculate the Spark query plan before r...

Data Engineering
Reply
  • 832 Views
  • 0 replies
  • 0 kudos
Labels