Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2024 05:22 AM - edited 11-21-2024 05:30 AM
The answers here are not correct.
TLDR: _After_ the Spark DF is materialized, saveAsTable takes ages. 35seconds for 1million rows.
saveAsTable() is SLOW - terribly so. Why? Would be nice to get an answer. The workaround is to avoid spark for delta - note I am not using Photon out of reasons. So just writing plain parquet with pyarrow.parquet and then read them with a SQL warehouse into a delta table (using Photon).
I have a tiny arrow data frame with 19 columns and 1million rows. The whole computation takes 2 seconds in stupid python and
spark_df = spark.createDataFrame(data.to_pandas())
spark_df.display()
take 1 second.
Then comes
spark_df.write.format("delta").mode("append").saveAsTable("default.hello_sleepy")
with a whopping 35 seconds?! What is that? Running this single threaded with delta-io writes instantly. Also pyarrow.parquet.write_table take a second. But saveAsTable 35? What is going on here?
When it is figured out to run the calculation equally fast single threaded on Databricks Spark as on a Raspberry Pi - then I would like to run this on worker executors for 15000 files in parallel. Actually this whole exercies might be better done in Lambda, but still it should be possible.
What am I missing?
What am I missing?
