cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Trying to write to S3 bucket but executed code not showing any progress

Megan05
New Contributor III

I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks under utilized. Any suggestions?

image

4 REPLIES 4

Vivian_Wilfred
Honored Contributor
Honored Contributor

Hi @Megan Finley​ , does this happen only while writing data to S3 or does it happen with all notebook commands? We usually see this behavior when the driver is not able to connect to workers. The cluster might be in a bad state. Please try restarting the cluster and check if it resolves this.

Hi Vivian,

This only happens with this command. I have tried attaching different clusters/restarting. The logs also say it is healthy.

User16753725469
Contributor II

Can you please check the driver log4j to see what is happening?

Hi @Megan Finley​,

Just a friendly follow-up. Did you check the driver's logs? did you try to save the data without the Python variables? did you try to do a mount on this location and then write?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!