- 4881 Views
- 1 replies
- 2 kudos
JDBC connector seems to be a bottleneck when trying to insert dataframe to Azure SQL Server
Hi! I am inserting a pyspark dataframe to Azure sql server and it takes a very long time. The database is a s4 but my dataframe that is 17 million rows and 30 columns takes up to 50 minutes to insert.Is there a way to significantly speed this up? I a...
- 4881 Views
- 1 replies
- 2 kudos
- 2 kudos
@Hjalmar Friden​ :There are several ways to improve the performance of inserting data into Azure SQL Server using JDBC connector:Increase the batch size: By default, the JDBC connector sends data in batches of 1000 rows at a time. You can increase th...
- 2 kudos
- 2947 Views
- 4 replies
- 0 kudos
Resolved! Datatype mismatch while reading data from sql server to databricks
Data from Azure sql server was read into databricks through JDBC connection (spark version 2.x) and stored into Gen1. Now the client wants to migrate the data from Gen1 to Gen2. When we ran the same jobs that read data from Azure Sql Server to Databr...
- 2947 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Mani Teja G​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
- 0 kudos