How to use Serverless as DBT-CLI compute?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2024 01:58 PM - edited 06-23-2024 01:58 PM
Hi,
we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.
I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine.
However, when executing the same dbt-run command while using serverless as DBT CLI I get the following error message:
Encountered an error:
Runtime Error
Database Error
HTTPSConnectionPool(host='my_host', port=443): Max retries exceeded with url: my_warehouse_url (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))
The Databricks serverless documentation mentions such errors:
"Python clients that use Databricks endpoints may encounter SSL verification errors such as “CERTIFICATE_VERIFY_FAILED”. To work around these errors, configure the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt. For example, run the following command at the beginning of a serverless notebook or job: import os; os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'"
I tried exactly this and added a task in the same job before the DBT-task and executed the python part mentioned above, but this didn't solve the problem.
Did anyone manage to use Serverless as DBT-CLI?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2024 01:10 AM
Hi,
thanks for your help!
Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.
I already tried adding the code you posted as a task before the DBT-task. That didn't help. I currently see no way of making the python code run directly in the DBT-task.
Any other ideas? 🙂

