Databricks

tlecomte · 01-02-2023

Dear Databricks community,I am using Spark Structured Streaming to move data from silver to gold in an ETL fashion. The source stream is the change data feed of a Delta table in silver. The streaming dataframe is transformed and joined with a couple ...

tlecomte · 08-01-2023

This blog post is the answer to my question: Adaptive Query Execution in Structured Streaming | Databricks BlogIn summary: the need is confirmed, and Databricks Runtime 13.1 enables AQE by default in foreachBatch sinks in non-Photon clusters. Looking...

tlecomte · 01-03-2023

Good question. In the nominal scenario the streams come from change data capture records from the data sources. These batches are usually small, with spikes to ~10k records when bulk operations occur at the data source. Nevertheless, even when the ba...

tlecomte · 01-02-2023

Thanks debayan. SPARK-17556 looks a bit stale. There are 2 linked pull requests that are both closed but not merged, and I could not find the files from the stacktrace above in any of these 2. So it looks like the code that runs in Databricks Spark i...

Databricks

User Stats

User Activity

Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch

Re: Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch

Re: Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch

Re: Enabling Adaptive Query Execution and Cost-Based Optimizer in Structured Streaming foreachBatch