Hey @I-am-Biplab ,

If running locally, it is going to be difficult to tune the performance up that much, but there are a few things you can try:

1. Up the partitions and batch size, as much as your machine will allow. Also, running repartition() could help ensure the data is actually partitioned.

2. You can up the SQL Warehouse size in case the throughput on that is the bottleneck. You could up it, write the data and then vertically scale it back down.

3. If you're writing from files, you could copy the files from your local machine to a Volume using the CLI, and then process the data from there which will allow more parallelized writes with Spark.

The bottleneck could be your machine configs, so I would also evaluate your architecture and see if there's any opportunities to stage the data in it's original format in Databricks or cloud storage first, since JDBC isn't best suited for large scale writes.