"Lakeflow helps our team level up. I don’t need to spend another three months training and coaching people – everything is unified in one framework."
- Dheeraj Puli, Head of Data Reliability Engineering, National Australia Bank
National Australia Bank (NAB) is modernizing its enterprise data platform by standardizing on Spark Declarative Pipelines with Databricks Lakeflow. By replacing hand-written Spark and legacy ETL with a more consistent, declarative approach, NAB is simplifying pipeline development at scale, improving reliability, and moving closer to a streaming-first architecture across the bank.
Key highlights:
- 1,800 Spark Declarative Pipelines running on Databricks Lakeflow: NAB is scaling a common framework across a large enterprise data estate used by 300–400 engineers.
- Job success rate improved from 86% to 99.6%: Standardization and built-in reliability are helping NAB run pipelines more consistently in production.
- 80% less transformation complexity: Some pipelines were reduced from highly complex custom logic to a much simpler declarative design, lowering operational risk and cognitive load.
- Faster path to streaming data: NAB has already standardized 100% of Bronze pipelines declaratively and migrated roughly 50% of Silver, with a long-term goal of end-to-end streaming from Bronze to Gold.
- Simpler onboarding and operations: Instead of training teams on multiple patterns and custom frameworks, Lakeflow gives engineers one consistent way to build and operate Spark pipelines.
🔗 Check out the full story here 👈
