04-08-2025 11:21 AM
Have you looked in the Spark UI to see exactly where the bottlenecks are? Also, what is the format of the data you are writing? e.g. CSV, Parquet, Delta?
