Hi @rt-slowth,
- Where are the tables and materialized views specified in Unity Catalog stored, in DBFS or metastore?
The tables and materialized views specified in Unity Catalog are stored in the metastore of the Azure Databricks workspace. You can use the CREATE EXTERNAL LOCATION statement to create a reference to a storage location in an Azure data lake Storage Gen2 container in your Azure account. You can also use the CREATE EXTERNAL TABLE or CREATE EXTERNAL VIEW statements to create a table or view that is backed by an external location. For more information, see Create External Locations and Create External Tables.
- Can I delete a parquet that has been readStreamed even once in S3?
Yes, using the spark, you can delete a parquet that has been readStreamed even once in S3.delete() method on the DataFrame returned by spark.readStream().
For more information, see Delete data from Delta Lake tables.
- Is there a way to save a DataFrame with Join and Window operations on a table read with dlt.read from a streaming Delta Live Table as a Table instead of a Materialized View?
Yes, there is a way to save a DataFrame with Join and Window operations on a table read with dlt.read from a streaming Delta Live Table as a Table instead of a Materialized View.
Based on the same query, you can use the @Dlt.table decorator to define both materialized views and streaming tables.
For more information, see Transform data with Delta Live Tables.
- The output of the @Dlt.table decorator seems to be created as a Materialized View, but is it possible to change it to a Table?
Yes, it is possible to change the output of the @Dlt.table decorator from being created as a Materialized View to being created as a Table.
You can use the @Dlt.view decorator instead of the @Dlt.table decorator for your function definition.
For more information, see Transform data with Delta Live Tables.
I hope this answers your questions. If you have any other questions about AWS DMS, Delta Live Tables, or Unity Catalog, please feel free to ask me. 😊