Display, count and write commands stuck after 1st job

[magy]

Hi,

I have problems with displaying and saving a table in Databricks.

Simple command can run for hours without any progress..

Before that I am not doing any rocket science - code runs in less than a minute, I have one join at the end.

I am using 7.3 LTS ML GPU cluster with Standard_NC12 worker and driver.

Dataset has around 3 mln rows.

Thanks in advance for any help!

[-werners-]

hard to tell without knowing how df_out is created.

As spark is lazy evaluated, the code is executed only at the write.

(transformations vs actions).

[Ryan_Chynoweth]

Agree with @werners here. If you share a screenshot of the execution plan then we may be able to help more.

One guess would be that you might need more partitions but I cannot be certain.

[jose_gonzalez]

hi @Just Magy​ ,

what is your data source? what type of lazy transformation and actions do you have in your code? Do you partition your data?

Please provide more details.