Hi @YS1 , Databricks Delta Lake and Delta Live Tables can be used to create a live dashboard for visualizing changes in streaming data.
- Delta Live Tables are designed for growing datasets and handle each row only once.
- Delta Lake can be used as a streaming source and sink, allowing you to process both existing and new data in a streaming query.
- Delta Lake's Change Data Capture (CDC) feature records changes to a Delta table, including updates and deletes.
- You can load a Delta table as a stream source using the provided code example.
- You can write data into a Delta table using Structured Streaming with the provided code example.
- If the schema of a Delta table changes during a streaming read, the query will fail, but it can be resolved by restarting the stream.