DBT Jobs || API call returns "Internal Error"

[dcc]

Hey there,

I am currently using the Databricks API to trigger a specific DBT job. For this, I am calling the API in a Web Activity on Azure datafactory and sending as headers the token and for the body I am sending the Job ID and the necessary vars I need for DBT.

Sometimes I get a "Internal Error", I cannot replicate the error since I don't know the cause, but when I do a retrial it works perfectly.

More details: I was using 2.1 version of the API, I already tried downgrading for 2.0 still get the same error sometimes. Also, I don't believe the problem is related to the body/headers since the retrial works and uses the exact same body/headers.

What could be a possible solution for this issue? Any workarrounds I can use to make sure this error does not occur?

[Kaniz_Fatma]

Hi @dcc, When encountering an “Internal Error” while using the Databricks API, it can be frustrating due to its vague nature.

However, let’s explore some potential solutions and workarounds:

Refresh and Restart:

• Sometimes, refreshing your browser page or restarting the SQL warehouse can resolve intermittent issues. If you’re using a web interface, try refreshing the page.
• Ensure that you’re not encountering any browser-related issues.
• Verify if the error persists after refreshing or restarting.

Check Dependencies:

• Double-check your code and configuration to ensure there are no issues specific to your implementation.
• Verify that all required dependencies are available and correctly set up in the Databricks environment.
• Retry the execution to see if the error persists.

Log Out and Log In:

• If you’re using Azure services, log out of Microsoft Azure completely.
• Visit portal.azure.com and log out.
• When you log back in, you should get the prompt to use Multi-Factor Authentication (MFA) to log in.
• If that doesn’t work, try logging out from all Azure services before attempting to log in again.

Cluster Considerations:

• Consider using a job cluster instead of an interactive cluster.
• Configure context auto-eviction (for AWS or Azure) to remove idle execution contexts.
• Use fewer notebooks

[jose_gonzalez]

Could you please share the driver logs? it will help us to narrow down the issue