What was the established architectural pattern for doing streaming ETL with Delta Lake before DLT was a thing? And incidentally, what approach would you take in the context of delta-oss today? The pipeline definitions would not have had to be declarative (as in DLT), but just in general.

One solution I am aware of is to e.g. rely on structured streaming and Trigger.Once in combination with an external orchestrator to execute the processing steps between the layers in delta. But what I'm interested in are use cases with end-to-end (bronze -> silver -> gold pipeline) latencies of less than a minute. This rules out at least some orchestrators.

So to summarize:

• A streaming pipeline with bronze, silver and gold tables
• End-to-end latency in the order of seconds (not sub-second, but less than a minute). Trigger.Once and batching with the help of an external orchestrator is not applicable if it results in higher latencies
• The use case would be "operational" i.e. the pipeline would have to be monitorable, recoverable and resumable if failed, debuggable, testable etc. even if it would require non-trivial amount of framework development

How did/do people tackle these types of scenarios?