tsilverstrim
New Contributor II

Hi Tony... there are several ways to accomplish this based on the non-functional requirements of your Use Case.  What does near real-time mean from a standpoint of signal to activation time from when the data is present in Databricks to when the activation via, for example, Streaming Destination should occur (not to get to deep in the weeds, but also consider the SLT... are you looking for 75%, 95%, 99% within the target time).  This will drive the design and the short answer is that converting Materialized data in a Database into a Stream and vice versa would occur upstream of AEP and suggest looking at https://docs.databricks.com/aws/en/dlt/what-is-change-data-capture , One suggestion is to also implement a full sync capability... issues can emerge and having a "get to a known data state" capability is prudent for high value/critical data and can dramatically reduce outage duration.