Databricks Community

james_

I am finding low worker node utilization when using Spatial SQL features. My cluster is DBR 17.1 with 2x workers and photon enabled.When I view the cluster metrics, they consistently show one worker around 30-50% utilized, the driver around 15-20%, a...

james_

Thank you again, @-werners- . I have a lot still to learn about partitioning and managing spatial data. Perhaps I mainly need more patience!

james_

In case anyone else stumbles here, I think I had my partitioning the wrong way around above - going from more partitions to less fixed the issue.

james_

Thank you for your reply, @-werners- . It turns out that partitioning was the issue, I changed it from ~2,500 to ~61,000 partitions (I think!) and it wrote in about half an hour. The partitions are very skewed, I haven't found a neat way to partition...

Databricks Community

User Stats

User Activity

Low worker utilisation in Spatial SQL

Re: Low worker utilisation in Spatial SQL

Re: Low worker utilisation in Spatial SQL

Re: Low worker utilisation in Spatial SQL