topic Display, count and write commands stuck after 1st job in Data Engineering

Display, count and write commands stuck after 1st job

magy — Fri, 15 Oct 2021 11:33:36 GMT

Hi,

I have problems with displaying and saving a table in Databricks.

Simple command can run for hours without any progress..

Before that I am not doing any rocket science - code runs in less than a minute, I have one join at the end.

I am using 7.3 LTS ML GPU cluster with Standard_NC12 worker and driver.

Dataset has around 3 mln rows.

Thanks in advance for any help!

-werners- — Fri, 15 Oct 2021 11:47:38 GMT

hard to tell without knowing how df_out is created.

As spark is lazy evaluated, the code is executed only at the write.

(transformations vs actions).

Ryan_Chynoweth — Fri, 15 Oct 2021 19:52:18 GMT

Agree with @werners here. If you share a screenshot of the execution plan then we may be able to help more.

One guess would be that you might need more partitions but I cannot be certain.

jose_gonzalez — Mon, 18 Oct 2021 22:31:46 GMT

hi @Just Magy ,

what is your data source? what type of lazy transformation and actions do you have in your code? Do you partition your data?

Please provide more details.