cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Display, count and write commands stuck after 1st job

magy
New Contributor

Hi,

I have problems with displaying and saving a table in Databricks.

Simple command can run for hours without any progress..

imageBefore that I am not doing any rocket science - code runs in less than a minute, I have one join at the end.

I am using 7.3 LTS ML GPU cluster with Standard_NC12 worker and driver.

Dataset has around 3 mln rows.

Thanks in advance for any help!

3 REPLIES 3

-werners-
Esteemed Contributor III

hard to tell without knowing how df_out is created.

As spark is lazy evaluated, the code is executed only at the write.

(transformations vs actions).

Ryan_Chynoweth
Honored Contributor III

Agree with @werners here. If you share a screenshot of the execution plan then we may be able to help more.

One guess would be that you might need more partitions but I cannot be certain.

jose_gonzalez
Moderator
Moderator

hi @Just Magy​ ,

what is your data source? what type of lazy transformation and actions do you have in your code? Do you partition your data?

Please provide more details.