Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Share Standard Delta Table -Instead of sharing the streaming table directly, share a regular Delta table with history that captures the same data. You can create a standard Delta table from the streaming source and share itWITH HISTORY to perform time travel. You can keep your streaming table for real time & run scheduled job to load the batch table that could be used by recipients.
Enable Change Data Feed (CDF) on a standard table - You can share a regular Delta table with CDF enabled for incremental history updates. Recipients can access row-level changes and merge them into a local copy if required. It reduces egress costs since only changes cross boundaries.
Materialized Views - You can create a materialized view on top of streaming table and periodically refresh the materialized view to capture snapshots. Its Less of real time but provides historical snapshots
