cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to increase timeout in Databricks Workflows DBT task

Mattias
New Contributor II

Hi,

I have a Databricks Workflows DBT task that targets a PRO SQL warehouse. However, the task fails with a "to many retries" error (see below) if the PRO SQL warehouse is not up and running when the task starts. How can I increase the timeout or allowed number of retries (for the DBT task connecting to the PRO SQL Warehouse)?

Error message:

12:03:16 Finished running in 0 hours 5 minutes and 4.37 seconds (304.37s).
12:03:16 Encountered an error: Database Error HTTPSConnectionPool(host='adb-5482477729738913.13.azuredatabricks.net', port=443): Max retries exceeded with url: /sql/1.0/warehouses/d426dd371415ad72 (Caused by ResponseError('too many 503 error responses'))

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

As per API doc for jobs you can set up a time out for your jobs: https://docs.databricks.com/api/workspace/jobs/create#tasks-timeout_seconds 

An optional timeout applied to each run of this job task. A value of 0 means no timeout.

Mattias
New Contributor II

Thanks for your response.

Is there a similar time out parameter when cofiguring a workflow in the Databricks Web GUI? I cannot find one...

Mattias
New Contributor II

One option seems to be to reference a custom "profiles.yml" in the job configuration and specify a custom DBT Databricks connector timeout there (https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#additional-parameters).

However, then I must also set the "token"-property manually, and I don't want to check in the token into git. Is there a way to solve authentication when using a custom profiles.yml, without checking in the Databricks token into git?