Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2022 01:30 PM
Spark, by default, works in a distributed way and does the process in parallel using native spark.
It will use all cores; every core will process one partition and write it to the database.
The same with selec.
So for example you can use the below code to read SQL in parallel:
table = (spark.read
.format("jdbc")
.option("url", "<jdbc_url>")
.option("dbtable", "<table_name>")
.option("user", "<username>")
.option("password", "<password>")
# a column that can be used that has a uniformly distributed range of values that can be used for parallelization
.option("partitionColumn", "<partition_key>")
# lowest value to pull data for with the partitionColumn
.option("lowerBound", "<min_value>")
# max value to pull data for with the partitionColumn
.option("upperBound", "<max_value>")
# number of partitions to distribute the data into. Use sc.defaultParallelism to set to number of cores
.option("numPartitions", sc.defaultParallelism)
.load()
My blog: https://databrickster.medium.com/