Pipelines using dlt modules from the Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2024 04:01 PM
[Situation]
I am using AWS DMS to store mysql cdc in S3 as a parquet file.
I have implemented a streaming pipeline using the DLT module.
The target destination is Unity Catalog.
[Questions and issues].
- Where are the tables and materialized views specified in Unity Catalog stored, in DBFS or metastore?
- Can I delete a parquet that has been readStreamed even once in S3?
- Is there a way to save a DataFrame with Join and Window operations on a table read with dlt.read from a streaming Delta Live Table as a Table instead of a Materialized View?
- The output of the @Dlt.table decorator seems to be created as a Matarialized View, but is it possible to change it to a Table?
You can answer them one by one.
- Labels:
-
Workflows