How do you keep a plain Delta copy of a streaming table? CLONE is not supported on streaming tables

[binlogreader]

Hi All, 

I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. 

My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep or shallow clone, and testing confirms it:

```
[STREAMING_TABLE_OPERATION_NOT_ALLOWED.UNSUPPORTED_OPERATION]
The operation DEEP CLONE is not allowed: The operation is not supported on Streaming Tables.
```

My questions:

1. Is there a supported mechanism I'm missing that produces an incremental (or even full), file-level copy of a streaming table, the way deep clone does for plain Delta tables?
2. If CDF plus MERGE is the intended pattern for this, is there guidance on doing it safely against a table that a pipeline owns (full refresh on the source, for example, changes the version history the copy job depends on)?
3. Is the clone limitation on streaming tables permanent by design, because the pipeline owns the table's files, or is it a current-state gap?

[adnan_alvee]

Have you looked into Compatibility Mode? It is explicitly built to generate a read-only compatibility version of streaming tables/materialized views at a chosen location and keep it refreshed automatically after updates.

[binlogreader]

Thanks @adnan_alvee , yes, we did test Compatibility Mode. 

The output is a copy of files at a storage location, not a Unity Catalog table.

Context: We are setting up an external engine (duckdb) to read UC Delta tables through credential vending, which keeps grants and auditing in one place. The compatibility copy is invisible to UC, so the reader would authenticate to the storage directly, and we would be maintaining both a second copy of the data and a second access-control path for it.

So Compatibility Mode answers "give an outside engine the bytes" but not "keep the read governed by Unity Catalog", and the second part is the real requirement behind my question. That's why the CDF plus MERGE pattern into a plain UC-registered table is currently our leading candidate despite the bookkeeping it carries.

If you know a way to have the compatibility output registered in UC, or if there are plans for it to produce a governed table rather than a location copy, that would close the gap completely and I'd gladly take that path.

[adnan_alvee · Accepted solution]

Then I think this feature could close the gap external data access for pipeline datasets 
It requires Delta 4.0+ via UC APIs support which DuckDB already supports.

Compatibility Mode vs External Data Access Mode

If that is not a path, then a CDF/MERGE copy into a plain UC table is still the practical workaround.

[binlogreader]

Streaming MV worked! You sir saved me a lot of time. Many thanks.