Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 12:06 PM
I understand you want it sooner. Did it at least write the data in 10 minutes compared to not writing before?
There are more knobs you can tweak like
spark.sql.shuffle.partitions=auto
Do you have any index columns in your spatial data that can be used for joining?
Also please check whether your data is partitioned correctly.
Finally, what is your rationale for stating that this should be completed in less than 10 minutes? Do you have anything to compare, or its just your feel.
~