Hubert-Dudek
Databricks MVP

Spark, by default, works in a distributed way and does the process in parallel using native spark.

It will use all cores; every core will process one partition and write it to the database.

The same with selec.

So for example you can use the below code to read SQL in parallel:

table = (spark.read
  .format("jdbc")
  .option("url", "<jdbc_url>")
  .option("dbtable", "<table_name>")
  .option("user", "<username>")
  .option("password", "<password>")
  # a column that can be used that has a uniformly distributed range of values that can be used for parallelization
  .option("partitionColumn", "<partition_key>")
  # lowest value to pull data for with the partitionColumn
  .option("lowerBound", "<min_value>")
  # max value to pull data for with the partitionColumn
  .option("upperBound", "<max_value>")
  # number of partitions to distribute the data into. Use sc.defaultParallelism to set to number of cores
  .option("numPartitions", sc.defaultParallelism)
  .load()


My blog: https://databrickster.medium.com/