Thank you. That's helpful.

One of the points of considering Databricks was to make use the elegance of the automated processes (schema detection), STORED AS SCD 2, etc. This could make for a clean code base and a more robust data flow.

However, at this rate, it looks like I would have to use traditional methodologies, so then the question arises - why bother with Databricks?

I'm investigating possibilities for post SCD processing but, so far, it's proving somewhat torturous...