- 5650 Views
- 6 replies
- 3 kudos
Resolved! Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch
Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...
- 5650 Views
- 6 replies
- 3 kudos
- 3 kudos
It's not recommended to have AQE on a Streaming query for the same reason you shared in the description. It has been documented here
- 3 kudos
- 4215 Views
- 1 replies
- 1 kudos
Resolved! ForeachBatch() - Get results from batchDF._jdf.sparkSession().sql('merge stmt')
Most python examples show the structure of the foreachBatch method as:def foreachBatchFunc(batchDF, batchId): batchDF.createOrReplaceTempView('viewName') ( batchDF ._jdf.sparkSession() .sql( ...
- 4215 Views
- 1 replies
- 1 kudos
- 1 kudos
Just found a solution...Need to convert the Java Dataframe (jdf) to a DataFramefrom pyspark import sql def batchFunc(batchDF, batchId): batchDF.createOrReplaceTempView('viewName') sparkSession = batchDF._jdf.sparkSession() resJdf = sparkSes...
- 1 kudos