cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use Serverless as DBT-CLI compute?

dbph
New Contributor II

Hi,

we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.

I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine.

However, when executing the same dbt-run command while using serverless as DBT CLI I get the following error message:

 

 

Encountered an error:
Runtime Error
  Database Error
    HTTPSConnectionPool(host='my_host', port=443): Max retries exceeded with url: my_warehouse_url (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))

 

The Databricks serverless documentation mentions such errors:

 

"Python clients that use Databricks endpoints may encounter SSL verification errors such as “CERTIFICATE_VERIFY_FAILED”. To work around these errors, configure the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt. For example, run the following command at the beginning of a serverless notebook or job: import os; os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'"

 

I tried exactly this and added a task in the same job before the DBT-task and executed the python part mentioned above, but this didn't solve the problem.

Did anyone manage to use Serverless as DBT-CLI?

Thanks

1 REPLY 1

dbph
New Contributor II

Hi, 

thanks for your help!

Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.

I already tried adding the code you posted as a task before the DBT-task. That didn't help. I currently see no way of making the python code run directly in the DBT-task.

Any other ideas? 🙂

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now