Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges

[vamsi_simbus]

Hi All,

I’m currently working on a Proof of Concept (POC) to migrate existing Talend ETL jobs to Databricks. The goal is to leverage Databricks for data processing and orchestration while moving away from Talend.

I’d appreciate insights on the following:

1. Migration Approach:

   • Is there a recommended strategy for converting Talend jobs (which use components like tMap, tFileInputDelimited, etc.) into Databricks workflows?
   • Should we rewrite logic using PySpark/SQL in notebooks, or is there any automation tool or accelerator available?

2. Data Orchestration:

   • How do you typically handle job scheduling and dependencies in Databricks compared to Talend’s job orchestration?
   • Any tips for integrating with Airflow or Databricks Workflows?

3. Performance & Optimization:

   • What are the best practices for optimizing ETL logic when moving from Talend’s row-based processing to Spark’s distributed architecture?

4. Common Pitfalls:

   • What challenges should we anticipate during migration (e.g., error handling, schema evolution, incremental loads)?

If anyone has gone through a similar migration or has resources, templates, or accelerators to share, that would be extremely helpful.

Thanks in advance!

[AbhaySingh]

Migrating from Talend’s jobs to Databricks requires rebuilding ETL logic using Spark. There is no native one-click converter provided by Databricks, so the typical approach is to audit and refactor each Talend job into code (PySpark or Spark SQL) in Databricks notebooks or scripts.

In terms of accelerators,  this blog may be of some help.

https://blog.aidetic.in/replaced-multiple-etl-tools-like-talend-with-databricks-5841e87bdaea

Also, saw someone providing a tools based migration service but use your discretion to engage with them - I haven't had any interaction with this service provider so can't endorse them.

https://travinto.com/migration-services/talend-to-databricks