Hey, Ivan!
Happy to share an area that we worked on recently.
Our team embarked on a transformative effort for one of our flagship products, aimed at enhancing our data processing capabilities, primarily to optimize execution performance and fully leverage the advanced features offered by the Unity Catalog. The heart of this effort was the migration of our ELT (Extract, Load, Transform) workloads from a combination of Azure Data Factory (ADF) and Notebooks to full Databricks Workflows orchestration. This move was motivated by the need to improve operational efficiency, reduce latency and capitalize on the Unity Catalog's out-of-the-box capabilities, such as column-level lineage of the delta tables. I would gladly share the challenges we faced, the solutions we implemented, and the significant performance improvements we achieved. Looking forward to share more if the topic sparks interest!
