topic Re: How to reduce time while loading data into the azure synapse table? in Data Engineering

How to reduce time while loading data into the azure synapse table?

tinendra — Tue, 14 Feb 2023 12:19:07 GMT

Hi All,

I just wanted to know if is there any option to reduce time while loading Pyspark Dataframe into the Azure synapse table using Databricks.

like..

I have a pyspark dataframe that has around 40k records and I am trying to load data into the azure synapse table using databricks it is taking almost 1.10 hrs+ to load complete data into the azure table. I am using save mode('overwrite') as per requirements.

Please let me know if any possible solution to reduce time.

Thanks,

Tinendra

Re: How to reduce time while loading data into the azure synapse table?

Ajay-Pandey — Tue, 14 Feb 2023 12:22:45 GMT

Hi @Tinendra Kumar ,

You can increase the DTU in synapse and if possible, use append mode while saving the files that will help you to reduce the time.

Re: How to reduce time while loading data into the azure synapse table?

tinendra — Tue, 14 Feb 2023 12:26:53 GMT

Hi @Ajay Pandey

I don't have any control over the azure side. Could you please tell me if is there any way/option to do this on the spark/databricks side?

Re: How to reduce time while loading data into the azure synapse table?

-werners- — Tue, 14 Feb 2023 13:04:30 GMT

have you checked this:

https://learn.microsoft.com/en-us/azure/databricks/archive/azure/synapse-polybase

tbh I do not use Databricks to load data into synapse. I write the data as parquet /delta lake on our data lake, and use ADF to copy to synapse if necessary. this goes pretty fast.

Another option is to use Synapse Serverless or External tables on the parquet files themselves.

Re: How to reduce time while loading data into the azure synapse table?

Ajay-Pandey — Tue, 14 Feb 2023 13:25:08 GMT

Hi @Tinendra Kumar ,

There is no option to check your permission in databricks.

Re: How to reduce time while loading data into the azure synapse table?

Anonymous — Fri, 17 Feb 2023 05:39:53 GMT

Hi @Tinendra Kumar

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!