Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-17-2024 11:08 PM
I set this properties to cluster level, but issue doesn't gets resolved
I am trying to read jdbc oracle table and write in unity catalog.
when i give high number of .option("numPartitions", partitions)\ like 100 or 50 to achieve maximum parallelism, then i get this heartbeat timed out issue
Cluster conf: i have (20 cores 140 GB) 5 min machines on my cluster with auto-scaling set to 10
but when i reduce this to num partitions 25, the issue doesn't occurs and everything runs fine
data is few tables with data around this 173313859
Any reasoning for this?