topic Re: Ingesting more than 7 million rows into a SQL Server Table in Data Engineering

Ingesting more than 7 million rows into a SQL Server Table

umair_hanif — Thu, 22 Jun 2023 12:12:45 GMT

Hi All, I hope you're super well.

I need your recommendations and solution for my problem.

I am using a Databricks instance DS12_v2 which has 28GB RAM and 4 cores. I am ingesting 7.2 million rows into a SQL Server table and it is taking 57 min - 1 hour 3 minutes just for ingesting the data.

Can you please recommend a solution so that I can decrease this to 35 mins or 40 mins with the existing hardware?

Re: Ingesting more than 7 million rows into a SQL Server Table

Anonymous — Fri, 23 Jun 2023 06:20:05 GMT

Hi @Umair Hanif

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Ingesting more than 7 million rows into a SQL Server Table

Anonymous — Tue, 27 Jun 2023 15:25:45 GMT

You can try to use BULK INSERT.

https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver16

Also using Data Factory instead of Databricks for the copy can be helpful.