Databricks Community

Agus1 · 11-08-2023

Hello!I'm using Structured Streaming to write to a delta table. The source is another delta table written with Structured Streaming as well. In order to datacheck the results I'm attempting to obtain from the checkpoint files of the target table the ...

Agus1 · 08-23-2023

I’m trying to implement a streaming pipeline that will run hourly using Spark Structured Streaming, Scala and Delta tables. The pipeline will process different items with their details.The source are delta tables that already exists, written hourly u...

Agus1 · 11-28-2023

Hello @Retired_mod, thank you for your answer.I'm a bit confused here because you seem to be describing the opposite behavior of what I've seen in our checkpoint files.Here I repost my examples to try to understand better.First checkpoint file:{"sour...

Agus1 · 08-24-2023

Hey @Mo I had the idea that stream to stream joins couldn't be performed when using "foreachBatch".Is this not the case?Also this part of the documentation specifies that joins can only be used with "append" mode.So it seems like it won't work with t...

Agus1 · 08-24-2023

Hello @Mo, thanks for the answer!I've considered using foreachBatch, but there are 2 issues:For some cases I need to use as source 2 streaming table and join them. I believe this is not supported by foreachBatch?Since I would be using merge for writt...

Databricks Community

User Stats

User Activity

Obtain the source table version number from checkpoint file when using Structured Streaming

Update destination table when using Spark Structured Streaming and Delta tables

Re: Obtain the source table version number from checkpoint file when using Structured Streaming

Re: Update destination table when using Spark Structured Streaming and Delta tables

Re: Update destination table when using Spark Structured Streaming and Delta tables