- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2024 04:24 PM
To the best of my knowledge DLT is a mini ETL system by its own. It took me quite a bit of time to get used to it as it is similar yet very different than your solution consisting of jobs and notebooks.
As it has ownership of the target "tables" (materialized views) it manages them and one step further inside Databricks' control and not in the hands of the developer.
I'm sure the objects are created in the managed storage but I don't know if you can easily find them or if they are meant to be found.
Once you delete the pipeline I assume that either the files are deleted straight away or after 7 days.
A somewhat convoluted way to find them in the storage layer is to have an empty Databricks environment where the first thing you create is a materialized view and this should narrow down the set of files you need to look through.