cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

DBT Jobs || API call returns "Internal Error"

dcc
New Contributor

Hey there,

I am currently using the Databricks API to trigger a specific DBT job. For this, I am calling the API in a Web Activity on Azure datafactory and sending as headers the token and for the body I am sending the Job ID and the necessary vars I need for DBT.

Sometimes I get a "Internal Error", I cannot replicate the error since I don't know the cause, but when I do a retrial it works perfectly.

More details: I was using 2.1 version of the API, I already tried downgrading for 2.0 still get the same error sometimes. Also, I don't believe the problem is related to the body/headers since the retrial works and uses the exact same body/headers.

What could be a possible solution for this issue? Any workarrounds I can use to make sure this error does not occur?

 

2 REPLIES 2

Kaniz
Community Manager
Community Manager

Hi @dcc, When encountering an “Internal Error” while using the Databricks API, it can be frustrating due to its vague nature.

 

However, let’s explore some potential solutions and workarounds:

 

Refresh and Restart:

  • Sometimes, refreshing your browser page or restarting the SQL warehouse can resolve intermittent issues. If you’re using a web interface, try refreshing the page.
  • Ensure that you’re not encountering any browser-related issues.
  • Verify if the error persists after refreshing or restarting.

Check Dependencies:

  • Double-check your code and configuration to ensure there are no issues specific to your implementation.
  • Verify that all required dependencies are available and correctly set up in the Databricks environment.
  • Retry the execution to see if the error persists.

Log Out and Log In:

Cluster Considerations:

  • Consider using a job cluster instead of an interactive cluster.
  • Configure context auto-eviction (for AWS or Azure) to remove idle execution contexts.
  • Use fewer notebooks

jose_gonzalez
Moderator
Moderator

Could you please share the driver logs? it will help us to narrow down the issue

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.