Spark dataframe performing poorly

[qwerty3]

I have huge datasets, transformation, display, print, show are working well on this data when read in a pandas dataframe. But the same dataframe when converted to a spark dataframe, is taking minutes to display even a single row and hours to write the data in a delta table.

[gchandra]

Can you please share the code snippet?

[qwerty3]

df.write.format("delta").saveAsTable("test_db.test_spark_df")

This took 3 mins to get completed with 5 rows and 4 columns

Actual datasets are not even getting written

[qwerty3]

df.write.format("delta").saveAsTable("test_db.test_spark_df")

This took 3 mins to get completed with 5 rows and 4 columns

Actual datasets are not even getting written

[gchandra]

3 mins to write 5 rows is no good.

Are you running this on a shared cluster with so many other jobs? Will it be possible to test this on a personal cluster to isolate the issue?

try displaying the data frame in one cell display(df) and save the data frame in another cell. 

[qwerty3]

The cluster that I was using to execute this was not performing any other tasks, although the azure quota for this cluster family cpu was 83% at the time, I created a new cluster belonging to a family which had all the cores available, there spark is working well. But even at 83% utilization, should that cluster (the one used earlier, with high memory) perform so poorly?

[gchandra]

It's good to hear it worked on the new cluster family.

If the quota is already at 83%, the number of nodes your cluster needs is important. If Azure is not able to provision that many resources, it could result in suboptimal performance.

To find out this, please reduce the number of nodes so your cluster can start the job and complete it.