topic How to enable AQE in foreachbatch mode in Data Engineering

How to enable AQE in foreachbatch mode

mjedy78 — Tue, 10 Dec 2024 08:33:34 GMT

I am processing the daily data from checkpoint to checkpoint everyday by using for each batch in streaming way.

df.writeStream.format("delta")
        .option("checkpointLocation", "dbfs/loc")
        .foreachBatch(transform_and_upsert)
        .outputMode("update")
        .trigger(availableNow=True)
        .start()

Due to skewness I want to enable aqe and set skewJoin optimization true

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.forceOptimizeSkewedJoin", "true")

However, when I checked the Spark UI settings, the value was set to false: spark.sql.adaptive.enabled = false.

I am using Databricks DBR 14.3.x-photon-scala2.12 with Photon enabled.

According to this https://www.databricks.com/blog/adaptive-query-execution-structured-streaming, AQE supports streaming for each batch query starting from DBR 13.2.

here is the settings in dataframe properties tab

Re: How to enable AQE in foreachbatch mode

MuthuLakshmi — Tue, 10 Dec 2024 12:21:54 GMT

@mjedy78
Did you set the config at cluster level or notebook level??
Can you try to set these config in cluster properties and check if that helps!

Re: How to enable AQE in foreachbatch mode

mjedy78 — Tue, 10 Dec 2024 13:01:27 GMT

I have tried both,
What I am triggering is a job, in job first I set within notebook level by adding spark.conf.set
Then I also added some configs in job cluster

Re: How to enable AQE in foreachbatch mode

mjedy78 — Tue, 10 Dec 2024 18:03:18 GMT

@MuthuLakshmi any idea?