jm99
New Contributor III

No I didn't. In fact I had to stop using DLT when another issue came up around performing a partial /streaming increment of a large platinum aggregation table. I ended up going back to using:

  • Kafka reader (see Consume Data From Apache Kafka )
  • Streaming dataframes
  • Delta table: enabling enableChangeDataFeed and processing the "readChangeFeed"
  • Using foreachBatch() method on the stream writer to apply the aggregation and required SCD1 upsert