cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Trying to write to S3 bucket but executed code not showing any progress

Megan05
New Contributor III

I am trying to write data from databricks to an S3 bucket but when I submit the code, it runs and runs and does not make any progress. I am not getting any errors and the logs don't seem to recognize I've submitted anything. The cluster also looks under utilized. Any suggestions?

image

4 REPLIES 4

Vivian_Wilfred
Databricks Employee
Databricks Employee

Hi @Megan Finleyโ€‹ , does this happen only while writing data to S3 or does it happen with all notebook commands? We usually see this behavior when the driver is not able to connect to workers. The cluster might be in a bad state. Please try restarting the cluster and check if it resolves this.

Hi Vivian,

This only happens with this command. I have tried attaching different clusters/restarting. The logs also say it is healthy.

User16753725469
Contributor II

Can you please check the driver log4j to see what is happening?

Hi @Megan Finleyโ€‹,

Just a friendly follow-up. Did you check the driver's logs? did you try to save the data without the Python variables? did you try to do a mount on this location and then write?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group