Databricks Community

ninjadev999 · 02-11-2022

I'm reading a huge csv file including 39,795,158 records and writing into MSSQL server, on Azure Databricks. The Databricks(notebook) is running on a cluster node with 56 GB Memory, 16 Cores, and 12 workers.This is my code in Python and PySpark:from ...

ninjadev999 · 02-11-2022

It's not solution what I want. sorry.I want to know what's wrong in my code and the reason why it was failed. and I want to improve the performance for big DataFrame processing.

ninjadev999 · 02-11-2022

I should make some changes in each records. After that, I write them into MSSQL. So looks like the Databricks is better to use it... I just needed to speed up to write big dataframe into MSSQL db and fix the error.

ninjadev999 · 02-11-2022

Thanks for your response!Actually, I don't know well about Data Factory.. To speed up to process big DataFrame, I used repartition mode, according to this blog: https://devblogs.microsoft.com/azure-sql/partitioning-on-spark-fast-loading-clustered-col...

Databricks Community

User Stats

User Activity

Can't write big DataFrame into MSSQL server by using jdbc driver on Azure Databricks

Re: Can't write big DataFrame into MSSQL server by using jdbc driver on Azure Databricks

Re: Can't write big DataFrame into MSSQL server by using jdbc driver on Azure Databricks

Re: Can't write big DataFrame into MSSQL server by using jdbc driver on Azure Databricks