Trying to write to S3 bucket but executed code not showing any progress

[Megan05]

I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks under utilized. Any suggestions?

[Vivian_Wilfred]

Hi @Megan Finley​ , does this happen only while writing data to S3 or does it happen with all notebook commands? We usually see this behavior when the driver is not able to connect to workers. The cluster might be in a bad state. Please try restarting the cluster and check if it resolves this.

[Megan05]

Hi Vivian,

This only happens with this command. I have tried attaching different clusters/restarting. The logs also say it is healthy.

[User16753725469]

Can you please check the driver log4j to see what is happening?

[jose_gonzalez]

Hi @Megan Finley​,

Just a friendly follow-up. Did you check the driver's logs? did you try to save the data without the Python variables? did you try to do a mount on this location and then write?