cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to debug Workflow Jobs timing out and DLT pipelines running forever?

espenol
New Contributor III

So I'm the designated data engineer for a proof of concept we're running, I'm working with one infrastructure guy who's setting up everything in Terraform (company policy). He's got the setup down for Databricks so we can configure clusters and run notebooks for reading/writing to ADLS Gen2 storage. However, I'm unable to run Workflows, Jobs time out after 10+ retries and DLT pipelines seems to run forever until I stop it.

I suspect the two issues are related. Do you have any tips on how to move forward? Could it be firewall? Some infrastructure improperly set up? On my own subscription without limitations I basically just use the portal to create workspace in 2 minutes and everything works fine. Here are the error message from the FIRST cell in the Job where it times out (yes it times out on the very first cell): log4j first, standard output second

https://gist.github.com/espenol/416609b4550bc682d375b03bb5d0619c

https://gist.github.com/espenol/b56b2ce08e5e2ed463141764ceeb5ef5

3 REPLIES 3

Pat
Honored Contributor III

Hi @Espen Solvang​ ,

I don't see much from the error except that it fails after X retries.

Do you have an error from the workflow UI?

thanks,

Pat

espenol
New Contributor III

It seems the problem resolved itself when we opened up a ton of communication that was being stopped by the firewall. The UI would only show that it was "Setting up tables".

Thanks for responding!

shan_chandra
Databricks Employee
Databricks Employee

@Espen Solvang​ - Just thought of checking with you, could you please let us know if you require further assistance on this?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group