cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Trying to write to S3 bucket but executed code not showing any progress

Megan05
New Contributor III

I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks under utilized. Any suggestions?

image

4 REPLIES 4

Vivian_Wilfred
Honored Contributor
Honored Contributor

Hi @Megan Finley​ , does this happen only while writing data to S3 or does it happen with all notebook commands? We usually see this behavior when the driver is not able to connect to workers. The cluster might be in a bad state. Please try restarting the cluster and check if it resolves this.

Hi Vivian,

This only happens with this command. I have tried attaching different clusters/restarting. The logs also say it is healthy.

User16753725469
Contributor II

Can you please check the driver log4j to see what is happening?

Hi @Megan Finley​,

Just a friendly follow-up. Did you check the driver's logs? did you try to save the data without the Python variables? did you try to do a mount on this location and then write?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.