topic Migrating huge table from synapse to databricks in Warehousing & Analytics

Migrating huge table from synapse to databricks

sruthianki — Thu, 21 Nov 2024 15:55:51 GMT

Hi ,

We are looking for a option to copy tables more than 50 TB to be copied from syanpse to databricks on weekly basis , please suggest of there are any feasible ways for same

we are using connector but it is taking too long to copy

https://learn.microsoft.com/en-us/azure/databricks/connect/external-systems/synapse-analytics

Re: Migrating huge table from synapse to databricks

-werners- — Thu, 21 Nov 2024 16:05:28 GMT

I suppose this is a Dedicated Sql pool you are talking about?
If so, I'd use Data Factory to extract the data from synapse and land it in ADLS in parquet format f.e. Then create a UC/Hive table on this.

Re: Migrating huge table from synapse to databricks

sruthianki — Thu, 21 Nov 2024 16:11:03 GMT

Thanks Warners.

Any documents related to same from databricks end which we can refer

Re: Migrating huge table from synapse to databricks

-werners- — Fri, 22 Nov 2024 08:25:03 GMT

There is no databricks documentation on this as it is only involved for a very tiny bit:
"CREATE TABLE catalog.schema.table USING PARQUET LOCATION 'url_to_the_parquet_files'.
All the rest is done in Azure Data Factory, or you can even use the built-in Pipelines of Synapse (in Synapse Studio).
There is a standard connector for Synapse and ADLS. The pipeline itself converts the data to parquet and writes is.
When done, do the create table on databricks.
Copy and transform data in Azure Synapse Analytics - Azure Data Factory & Azure Synapse | Microsoft Learn