Trying to write to S3 bucket but executed code not showing any progress

Megan05
New Contributor III

‎06-29-2022 09:04 AM

I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks under utilized. Any suggestions?

image

Labels:
Preview file
16 KB
0 Kudos

Vivian_Wilfred
Databricks Employee
Databricks Employee

‎06-29-2022 01:50 PM

Hi @Megan Finley​ , does this happen only while writing data to S3 or does it happen with all notebook commands? We usually see this behavior when the driver is not able to connect to workers. The cluster might be in a bad state. Please try restarting the cluster and check if it resolves this.

Megan05
New Contributor III
In response to Vivian_Wilfred

‎06-29-2022 01:55 PM

Hi Vivian,

This only happens with this command. I have tried attaching different clusters/restarting. The logs also say it is healthy.

0 Kudos

User16753725469
Databricks Employee
Databricks Employee

‎07-15-2022 06:40 AM

Can you please check the driver log4j to see what is happening?

0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee
In response to User16753725469

‎07-20-2022 04:07 PM

Hi @Megan Finley​,

Just a friendly follow-up. Did you check the driver's logs? did you try to save the data without the Python variables? did you try to do a mount on this location and then write?

0 Kudos