cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use Serverless as DBT-CLI compute?

dbph
New Contributor II

Hi,

we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.

I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine.

However, when executing the same dbt-run command while using serverless as DBT CLI I get the following error message:

 

 

Encountered an error:
Runtime Error
  Database Error
    HTTPSConnectionPool(host='my_host', port=443): Max retries exceeded with url: my_warehouse_url (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))

 

The Databricks serverless documentation mentions such errors:

 

"Python clients that use Databricks endpoints may encounter SSL verification errors such as “CERTIFICATE_VERIFY_FAILED”. To work around these errors, configure the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt. For example, run the following command at the beginning of a serverless notebook or job: import os; os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'"

 

I tried exactly this and added a task in the same job before the DBT-task and executed the python part mentioned above, but this didn't solve the problem.

Did anyone manage to use Serverless as DBT-CLI?

Thanks

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @dbph,

    • You can disable SSL certificate verification in Databricks CLI by specifying insecure = True in your Databricks configuration file (databrickscfg). This might help bypass the SSL error1.
    • The Databricks serverless documentation suggests configuring the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt.

To achieve this, you can run the following command at the beginning of your serverless notebook or job:

import os
os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'

dbph
New Contributor II

Hi, 

thanks for your help!

Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.

I already tried adding the code you posted as a task before the DBT-task. That didn't help. I currently see no way of making the python code run directly in the DBT-task.

Any other ideas? 🙂

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!