Ingesting more than 7 million rows into a SQL Server Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2023 05:12 AM
Hi All, I hope you're super well.
I need your recommendations and solution for my problem.
I am using a Databricks instance DS12_v2 which has 28GB RAM and 4 cores. I am ingesting 7.2 million rows into a SQL Server table and it is taking 57 min - 1 hour 3 minutes just for ingesting the data.
Can you please recommend a solution so that I can decrease this to 35 mins or 40 mins with the existing hardware?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2023 11:20 PM
Hi @Umair Hanif
Great to meet you, and thanks for your question!
Let's see if your peers in the community have an answer to your question. Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-27-2023 08:25 AM
You can try to use BULK INSERT.
Also using Data Factory instead of Databricks for the copy can be helpful.
