Hi,
we're exploring replacing one of the use cases we are running in our clour provider with a Databricks pipelines. We currently have explored possibility to subscribe to an eventhub using SDP pipelines, feedding our iot data into a Delta table where we keep full history.
This is impractical for usage in web applications for high latency. Our idea is to use Lakehouse synced table, however we don't want to sync with full history (too much data). Current idea is to have a dedicated delta table which would be filled by an SDP stream (to contain iot data near real-time), and then perhaps to be periodically cleaned to only contain last 7 days of data. That table would then be synced. There are all kinds of pitfalls  - combinining streaming ingestion with batch clean-up, syncing to Lakehose. Is this idea feasible? If not, how could these requirements be fullfiled?