every cpu process 1 partition at the time, other will wait. You can have autoscale like 2-8 executors every with 4 cpus, so it will process max 32 (4x8) partitions concurrently.

Please check also network configuration. Private link to connect to ADLS is recommended.

After df = spark.read.jdbc please verify partition number using df.rdd.getNumPartitions()