Structured Streaming Real-Time Mode Doesn’t Support Delta — What’s the Plan?

[rohan22sri]

Real-time mode doesn’t currently support Delta tables in Structured Streaming. Is there a planned timeline for launching this support?

Also, are there any plans for declarative pipelines to support real-time mode or provide an equivalent capability?

[amirabedhiafi]

Hello Rohan !

It is also one of the questions that I have in mind and I was checking the doc but unfortunately real time mode does not support delta tables as either a source or sink  and lakeflow spark declarative pipelines are also not supported in real time mode.

I tried to find any public ETA for it but in vain.

The closest declarative alternative today is lakeflow spark declarative pipelines with streaming tables in continuous mode which Databricks positions for freshness on the order of seconds to a few minutes and for sub-second latency, the current path is still structured streaming real time mode with supported sources such as kafka or event hubs rather than delta.

[rohan22sri]

Thanks @amirabedhiafi ,

Yes, achieving millisecond latency with Databricks is still not feasible when using Delta Lake tables. I’m also skeptical about whether true ms-level latency will be possible with Delta tables, given that writes themselves typically take a few seconds to commit.

I’m not sure if there are plans to address this limitation. Since most real-world use cases rely on Delta tables, having support for lower-latency or real-time capabilities would be highly practical and beneficial for many projects.

[amirabedhiafi]

Hi again !

Tere usually is not a single detailed public roadmap with dates for Databricks features. I think the best official places to track what is coming are release notes, preview or beta announcements the blog and the product feedback and ideas channel