Hubert-Dudek
Databricks MVP

Please increase parallelism by adjusting jdbc settings:

columnName="key",

lowerBound=1L,

upperBound=100000L,

numPartitions=100,

It is example values. The best that key column would be unique and continuous so it will be divided equally without data skews.

Please analyze also Spark UI - look what takes the biggest time (reading or writing?)


My blog: https://databrickster.medium.com/

View solution in original post