Lakeflow SDP (DLT) produce external tables, or only UC-managed
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
As I understand it, streaming tables and materialized views produced by Lakeflow Spark Declarative Pipelines (DLT) are always Unity Catalog managed tables , there's no LOCATION/path option on create_streaming_table or apply_changes.
Is that correct? A few questions:
Is there any supported way to make an SDP output a true external table?
If not, what are the recommended design patterns when you need the data as external — e.g. a designated
managed storage location at the catalog/schema level, a pipeline sink writing to an external path, or a downstream job that re-writes to CREATE TABLE ... LOCATION?
Any reason to prefer one pattern over another for a Gold SCD2 dimension?
Thanks!
Labels:
- Labels:
-
Spark