This is exactly how we implemented it in pySpark in 2018. It's our own solution, and it is flexible and robust compared to the third-party tools like Fivetran and Qlik. It saves millions of dollars in subscription fees.

Looking forward to the next parts.