cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Use of Genie agents for ETL migration

raman7292
New Contributor

I am currently working on a project to migrate existing ETL workflows from SSIS to Databricks. I would like to understand how Databricks Genie, Genie Code, or other Databricks AI-assisted capabilities can support this migration.

Can these tools analyze SSIS packages, including .dtsx files, control flows, data flows, transformations, variables, and configurations, and automatically generate equivalent Python, PySpark, or SQL code for Databricks?

For example, is it possible to upload or provide the SSIS package files and have Databricks generate production-ready notebooks or jobs that can be executed with minimal manual intervention?

I am particularly interested in understanding:

  • Which parts of an SSIS migration can realistically be automated?
  • Can Genie directly interpret .dtsx files, or must the SSIS logic first be exported into another format?
  • How well does it handle complex transformations, stored procedures, lookup components, error handling, and package dependencies?
  • How much manual validation, optimization, and redesign would still be required?
  • Are there any recommended tools, accelerators, or best practices for migrating SSIS workloads to Databricks?

I would appreciate hearing from anyone who has used Databricks AI-assisted tools for an SSIS migration or completed a similar project.

1 REPLY 1

adnan_alvee
Databricks Employee
Databricks Employee

Genie code can meaningfully accelerate an SSIS migration, but they should be viewed as a productivity layer around the migration process rather than a push-button solution. In practice, the most reliable automation today is around assessment, inventory, complexity analysis, SQL conversion where logic can be isolated cleanly, and acceleration of target-state pipeline development inside Databricks.

For SSIS specifically, the realistic approach is usually hybrid. Lakebridge can assess SSIS packages and help identify package structure, tasks, and complexity, but SSIS logic often needs to be broken apart first, especially where business logic is embedded in .dtsx XML, Execute SQL tasks, control-flow orchestration, variables, or custom script components. Once SQL and transformation logic are extracted and understood, Databricks tools can help convert parts of that logic and Genie Code can help engineers build or refine the corresponding Lakeflow pipelines, notebooks, SQL, and orchestration patterns on Databricks.

Complex transformations, stored procedures, lookups, package dependencies, error handling, and especially C#/VB script tasks typically require engineering review, Spark optimization, and source-to-target testing before the result is production-ready. The best practice is to use Databricks automation to speed up assessment, code generation, and validation, while planning for human review and modernization of the target architecture rather than aiming for a strict 1:1 conversion of every SSIS package.