gchandra
Databricks Employee
Databricks Employee

I understand you want it sooner. Did it at least write the data in 10 minutes compared to not writing before?

There are more knobs you can tweak like 

spark.sql.shuffle.partitions=auto

Do you have any index columns in your spatial data that can be used for joining?

Also please check whether your data is partitioned correctly.

Finally, what is your rationale for stating that this should be completed in less than 10 minutes? Do you have anything to compare, or its just your feel.

 



~