Ingesting more than 7 million rows into a SQL Server Table

[umair_hanif]

Hi All, I hope you're super well.

I need your recommendations and solution for my problem.

I am using a Databricks instance DS12_v2 which has 28GB RAM and 4 cores. I am ingesting 7.2 million rows into a SQL Server table and it is taking 57 min - 1 hour 3 minutes just for ingesting the data.

Can you please recommend a solution so that I can decrease this to 35 mins or 40 mins with the existing hardware?

[Anonymous]

Hi @Umair Hanif​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[Anonymous]

You can try to use BULK INSERT.

https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver1...

Also using Data Factory instead of Databricks for the copy can be helpful.