Databricks Community

Hi, I wonder what is the supposed to be the behavior of spark structured streaming when using the AvailableNow Trigger and there is a query failure during the query ? More specifically, what happens to the initial end offset set ? Does it change ? Wh...

Looking into the followinghttps://docs.databricks.com/en/structured-streaming/delta-lake.html#specify-initial-positionI am unclear as to what is the exact difference (if any) between "startingVersion: The Delta Lake version to start from. Databricks ...

Does the pre-partitioning of a Delta Table has an influence on the number of "default" Partition of a Dataframe when readying the data?Put differently, using spark structured streaming, when readying from a delta table, is the number of Dataframe par...

I would like to do a groupby followed by a join in structured streaming. I would read from from two delta table in snapshot mode i.e. latest snapshot.My question is specifically about chaining the stateful operator. groupby is update modechaning grou...