cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

tlecomte
by New Contributor III
  • 2309 Views
  • 6 replies
  • 3 kudos

Resolved! Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch

Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...

  • 2309 Views
  • 6 replies
  • 3 kudos
Latest Reply
Lingesh
New Contributor III
  • 3 kudos

It's not recommended to have AQE on a Streaming query for the same reason you shared in the description. It has been documented here

  • 3 kudos
5 More Replies
User16869510359
by Esteemed Contributor
  • 670 Views
  • 1 replies
  • 0 kudos

What are the best practices for Adaptive query execution

What are common configurations used and which workload will get benefit

  • 670 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Contributor
  • 0 kudos

Leave it turned on. the bet is with each Spark version released AQE will get better and better and eventually will lead to a much more performance optimisation plan than manually trying to tune it.

  • 0 kudos
User16826992666
by Valued Contributor
  • 560 Views
  • 1 replies
  • 0 kudos

Resolved! Do I still need to use skew join hints if I have Adaptive Query Execution enabled?

From what I have read about AQE it seems to do a lot of what skew join hints did automatically. So should I still be using skew hints in my queries? Is there harm in using them?

  • 560 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

With AQE Databricks  has the most up-to-date accurate statistics at the end of a query stage and can opt for a better physical strategy and or do optimizations that used to require hints,In the case of skew join hints, is recommended to rely on AQE...

  • 0 kudos
User16783855117
by Contributor II
  • 456 Views
  • 0 replies
  • 0 kudos

Is there a way to know if Adaptive Query Execution with Spark 3 has changed my Spark plan?

From the demo notebook located here (https://databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html) it seems like the approach to demonstrate AQE was working was to first calculate the Spark query plan before r...

  • 456 Views
  • 0 replies
  • 0 kudos
Labels