- 3580 Views
- 5 replies
- 4 kudos
Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection
I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...
- 3580 Views
- 5 replies
- 4 kudos
- 14099 Views
- 1 replies
- 0 kudos
ConnectException error
I'm using PySpark on Databricks and trying to pivot a 27753444 X 3 matrix. If I do it in Spark DataFrame: df = df.groupBy("A").pivot("B").avg("C") it takes forever (after 2 hours and I canceled it). If I convert it to pandas dataframe and then pivo...
- 14099 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Raymond_Hu,This means that the driver crashed because of an OOM (Out of memory) exception and after that, it's not able to establish a new connection with the driver. Please try below optionsTry increasing driver-side memory and then retry.You ca...
- 0 kudos