Databricks has shared a practical framework for ETL migration that moves the conversation away from one-size-fits-all rewrites. The core idea is simple: most teams do not choose a single migration path, they use a mix of Spark Declarative Pipelines, Lakehouse / Databricks SQL, and PySpark or Spark SQL notebooks depending on the workload.
Whatโs new
- Three paths, not one: Databricks frames migration as a tool-selection problem, with Spark Declarative Pipelines suited to managed ETL and data quality, Databricks SQL well suited to SQL-heavy workloads, and PySpark or Spark SQL notebooks reserved for more complex engineering logic.
- Phase the migration for outcomes: Rather than a big-bang cutover, the recommended approach is to assess first, capture quick wins, modernize the pipelines worth redesigning, and then optimize once the legacy platform is being retired.
- Use automation where it helps most: Lakebridge is positioned as a migration accelerator for profiling, assessment, SQL and ETL conversion, validation, and reconciliation, so teams can spend more time on business validation and less on mechanical translation.
- Start with low-risk, high-visibility wins: Databricks emphasizes using early migrations to build confidence, especially where simple SQL jobs or visible reporting flows can show value quickly.
- The end goal is simplification, not just conversion: The post makes the case that successful migrations retire old schedulers, redundant validation layers, and fragmented tooling instead of recreating them on the new platform.
Databricks also points to migrations like Walgreens, which moved off legacy Teradata, now processes about 40,000 data events per second, and uses the lakehouse for near real-time supply chain and pharmacy workflows.
๐ Read the full post here
