@ForestDD I could see a pre-installed mssql library available as part of DBR 13.3 LTS.https://docs.databricks.com/en/release-notes/runtime/13.3lts.html#:~:text=com.microsoft.sqlserver-,mssql,-%2Djdbc
Could you try using this and let us know if you ar...
@lawrence009 You can also take a look at the individual task level metrics. This should help in understanding whether there was skew involved during the processing. We can also get a better understanding of the spill by viewing the Task Level Summary...
@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false
@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false
