Databricks Community

Maatari

Assuming i need to perfom a groupby i.e. aggregation on a dataset stored in a delta table. If the delta table is partitioned by the field by which to group, can that have an impact on the suffling that the groupby would normally cause ? As a connecte...

Maatari

Does the pre-partitioning of a Delta Table has an influence on the number of "default" Partition of a Dataframe when readying the data?Put differently, using spark structured streaming, when readying from a delta table, is the number of Dataframe par...

Maatari

I would like to do a groupby followed by a join in structured streaming. I would read from from two delta table in snapshot mode i.e. latest snapshot.My question is specifically about chaining the stateful operator. groupby is update modechaning grou...

Maatari · 07-11-2024

I have just find out spark structured streaming do not commit offset to kafka but use its internal checkpoint system and that there is no way to visualize its consumption lag in typical kafka UI- https://community.databricks.com/t5/data-engineering/c...

Maatari · 05-19-2024

Hi, While ingesting files from a source folder continuously, I would like to be able to detect the case where files are being deleted. As far as I can tell the Autoloader can not handle the detection of files deleted in the source folder. Hence the c...

Maatari

I am confused about this recommendation. I thought the use of the append output mode in combination with aggregate queries is restricted to queries for which the aggregation is expressed using event-time and it defines a watermark.Could you clarify ?

Maatari · 05-14-2024

This `maxBytesPerTrigger` does not work for file source right ? Only source like Kafka Source ?

Databricks Community

User Stats

User Activity

Pre-Partitioning a delta table to reduce suffling of wide operation

Readying a partitioned Table in Spark Structured Streaming

Chaining stateful Operator

How to monitor Kafka consumption / lag when working with spark structured streaming?

DataBricks Auto loader vs input source files deletion detection

Re: Structured Streaming from a delta table that is a dump of kafka and get the latest record per ke

Re: Fixed interval micro-batches and AvailableNow Trigger