topic Re: Migrating data from synapse to databricks in Data Engineering

Migrating data from synapse to databricks

Krizofe — Sun, 18 Aug 2024 04:40:58 GMT

Hello team,

I have a requirement of moving all the table from Azure Synapse (dedicated sql pool) to databricks.

we have a data coming up from source to azure data lake frequently.

we have Azure data factory to load data (data flow does the basic transformation ) to Synapse.

we are looking to migrate data(tables) from synapse to databricks as a requirement.

And the new data from azure data lake to be ingested directly to databricks bypassing synapse.

can you please let me know the best approach to perform this action.

your insights are valuable!! Thanks

Re: Migrating data from synapse to databricks

Rishabh-Pandey — Sun, 18 Aug 2024 06:21:15 GMT

Use Azure Data Factory (ADF):

Configure a pipeline in ADF to copy data from Synapse SQL to Azure Data Lake. Set up an ADF Copy Activity to handle this data transfer.
Source Dataset: Azure Synapse SQL table.
Sink Dataset: Azure Data Lake Storage or Azure Blob Storage.
Data Movement Activity: Use the Copy Data activity in ADF.

Load Data into Databricks

Create Delta Tables in Databricks:
- Use Delta Lake for storing and managing data in Databricks. Delta Lake provides ACID transactions and supports efficient queries.
Automate Data Ingestion:
- Set up Databricks jobs or Delta Live Tables to automate the ingestion of data from Azure Data Lake into Delta tables.
Update Data Pipeline
1. Adjust ADF Pipelines:
  - Update your Azure Data Factory pipelines to load new data directly into Databricks instead of Synapse.
  - Source Dataset: Azure Data Lake Storage.
  - Sink Dataset: Databricks Delta table.

Re: Migrating data from synapse to databricks

Krizofe — Sun, 18 Aug 2024 06:37:37 GMT

Hello Rishabh Pandey,

Thanks for the reply but just have some concerns.

The data volume is huge like in billions and data gets deleted once loaded to Synapse.

Pushing all those data back to Azure Storage account from Synapse will be time consuming and costing more on storage. (Note: we have 3k tables with billions of rows)

So, we have any approach to directly move data to Databricks without staging it to Azure storage.

Thanks for the reply, mate.

Your insights will be valuable.

Re: Migrating data from synapse to databricks

Retired_mod — Mon, 19 Aug 2024 12:41:12 GMT

Hi @Krizofe, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!

Re: Migrating data from synapse to databricks

Rishabh-Pandey — Tue, 20 Aug 2024 04:36:12 GMT

@Krizofe You can use Azure Databricks to directly query and load data from Azure Synapse using Apache Spark. Databricks has built-in connectors that allow it to read from Synapse (or its underlying SQL Data Warehouse).

load it from synapse and then write it to ADLS .

Re: Migrating data from synapse to databricks

ebm — Wed, 16 Oct 2024 19:47:47 GMT

Hi Rishabh,

Thanks for the explanation, it looks a little oversimplified. I feel like copying tables from one place to another is not a big deal, however, the real challenge is to update the stored procedures and deal with system integrations. What is your recommendation for that?

Re: Migrating data from synapse to databricks

Stefan-Koch — Wed, 16 Oct 2024 20:15:28 GMT

Hi @Krizofe

Instead of copying all the data with ADF into files in the Data lake, you could use Lakehouse Federation in Databricks. Create a connection to Azure Synapse, and then you can query your data from Synapse directly in a catalog in Unity Catalog.

Re: Migrating data from synapse to databricks

thelogicplus — Sun, 24 Nov 2024 17:54:22 GMT

Hi @Krizofe , Just gone through you deatils and thought our similar experience with Azure Synapse to databrick migration. We faced a similar situation and were initially hesitant, One of the my colleague recommanded to use Travinto Technologies accelator, but after using Travinto Technologies' ETL migrator solution and services, we were blown away.Their accelerator helped us seamlessly migrate data and ETL jobs for our clients, saving us time and effort. We highly recommend Travinto Technologies.

topic Re: Migrating data from synapse to databricks in Data Engineering

Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks

Load Data into Databricks

Update Data Pipeline

Re: Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks

Re: Migrating data from synapse to databricks