I'm eagerly awating the next part. I do have some questions that I hope will get addressed, or maybe someone can share some insights on.
1. Are the framework tables stored as Delta tables in the lakehouse? I envision a seperate config schema that would house these tables, but I haven't worked with ingesting YAML files before so a bit confused on this point.
2. Will there be sample code showing a potential implementation of the generic execution notebooks? I am very much interested in how the audit tables get populated at run time with the statistics. We homebrewed an ETL monitoring system that relies on orchestration from Azure Data Factory, and I am hoping to be able to replace that with something more streamlined.
3. The article mentioned a separate pipeline to ingest the YAML files - is that planned for a future part of this series? I've tried searching the community forums but haven't been able to find much information on this, so I hope it is covered.
Excellent article, and I am excited to see what I can learn from the future parts so that I can apply them to my ETL environment.