How to increase timeout in Databricks Workflows DBT task
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-27-2025 05:57 AM
Hi,
I have a Databricks Workflows DBT task that targets a PRO SQL warehouse. However, the task fails with a "to many retries" error (see below) if the PRO SQL warehouse is not up and running when the task starts. How can I increase the timeout or allowed number of retries (for the DBT task connecting to the PRO SQL Warehouse)?
Error message:
12:03:16 Finished running in 0 hours 5 minutes and 4.37 seconds (304.37s).
12:03:16 Encountered an error: Database Error HTTPSConnectionPool(host='adb-5482477729738913.13.azuredatabricks.net', port=443): Max retries exceeded with url: /sql/1.0/warehouses/d426dd371415ad72 (Caused by ResponseError('too many 503 error responses'))
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-27-2025 06:10 AM
As per API doc for jobs you can set up a time out for your jobs: https://docs.databricks.com/api/workspace/jobs/create#tasks-timeout_seconds
An optional timeout applied to each run of this job task. A value of 0
means no timeout.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-27-2025 10:49 PM
Thanks for your response.
Is there a similar time out parameter when cofiguring a workflow in the Databricks Web GUI? I cannot find one...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-28-2025 12:45 AM
One option seems to be to reference a custom "profiles.yml" in the job configuration and specify a custom DBT Databricks connector timeout there (https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup#additional-parameters).
However, then I must also set the "token"-property manually, and I don't want to check in the token into git. Is there a way to solve authentication when using a custom profiles.yml, without checking in the Databricks token into git?

