Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2022 08:53 AM
No I didn't. In fact I had to stop using DLT when another issue came up around performing a partial /streaming increment of a large platinum aggregation table. I ended up going back to using:
- Kafka reader (see Consume Data From Apache Kafka )
- Streaming dataframes
- Delta table: enabling enableChangeDataFeed and processing the "readChangeFeed"
- Using foreachBatch() method on the stream writer to apply the aggregation and required SCD1 upsert