nlnsha
New Contributor II

I set this properties  to cluster level, but issue doesn't gets resolved

I am trying to read jdbc oracle table and write in unity catalog.

when i give high number of .option("numPartitions", partitions)\ like 100 or 50 to achieve maximum parallelism, then i get this heartbeat timed out issue

Cluster conf: i have (20 cores 140 GB) 5 min machines on my cluster with auto-scaling set to 10

but when i reduce this to num partitions 25, the issue doesn't occurs and everything runs fine

data is few tables with data around this 173313859

Any reasoning for this?