- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-27-2024 02:50 PM
Sure, and for the record I'm still not fully happy with how parameters need to be set at design time.
As mentioned, I store the metadata in a .json file that I read using a standard notebook. The content of that I then save into DBFS as a delta table overwriting any previous version. Then the DLT notebook reads from that table and I only need to specify the name of the process (e.g. IngestAdventureWorks) and that name matches the name of the DLT pipeline itself (or it can be derived).
Once I determine which table to read from the DLT pipeline can be driven by the metadata in that table.
I still find dealing with DLTs inconsistent with orchestration of standard notebook-driven data handling, it is an odd-one out that mostly needs a slightly different way of handling but so far I have found a workaround for every of these small inconsistencies.