topic How to increase timeout in Databricks Workflows DBT task in Data Engineering

How to increase timeout in Databricks Workflows DBT task

Mattias — Mon, 27 Jan 2025 13:57:16 GMT

Hi,

I have a Databricks Workflows DBT task that targets a PRO SQL warehouse. However, the task fails with a "to many retries" error (see below) if the PRO SQL warehouse is not up and running when the task starts. How can I increase the timeout or allowed number of retries (for the DBT task connecting to the PRO SQL Warehouse)?

Error message:

12:03:16 Finished running in 0 hours 5 minutes and 4.37 seconds (304.37s).
12:03:16 Encountered an error: Database Error HTTPSConnectionPool(host='adb-5482477729738913.13.azuredatabricks.net', port=443): Max retries exceeded with url: /sql/1.0/warehouses/d426dd371415ad72 (Caused by ResponseError('too many 503 error responses'))

Re: How to increase timeout in Databricks Workflows DBT task

Walter_C — Mon, 27 Jan 2025 14:10:45 GMT

As per API doc for jobs you can set up a time out for your jobs: https://docs.databricks.com/api/workspace/jobs/create#tasks-timeout_seconds

An optional timeout applied to each run of this job task. A value of 0 means no timeout.

Re: How to increase timeout in Databricks Workflows DBT task

Mattias — Tue, 28 Jan 2025 06:49:41 GMT

Thanks for your response.

Is there a similar time out parameter when cofiguring a workflow in the Databricks Web GUI? I cannot find one...

Re: How to increase timeout in Databricks Workflows DBT task

Mattias — Tue, 28 Jan 2025 08:45:46 GMT

One option seems to be to reference a custom "profiles.yml" in the job configuration and specify a custom DBT Databricks connector timeout there (https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#additional-parameters).

However, then I must also set the "token"-property manually, and I don't want to check in the token into git. Is there a way to solve authentication when using a custom profiles.yml, without checking in the Databricks token into git?