How to use Serverless as DBT-CLI compute?

[dbph]

Hi,

we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.

I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine.

However, when executing the same dbt-run command while using serverless as DBT CLI I get the following error message:

Encountered an error:
Runtime Error
  Database Error
    HTTPSConnectionPool(host='my_host', port=443): Max retries exceeded with url: my_warehouse_url (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))

The Databricks serverless documentation mentions such errors:

"Python clients that use Databricks endpoints may encounter SSL verification errors such as “CERTIFICATE_VERIFY_FAILED”. To work around these errors, configure the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt. For example, run the following command at the beginning of a serverless notebook or job: import os; os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'"

I tried exactly this and added a task in the same job before the DBT-task and executed the python part mentioned above, but this didn't solve the problem.

Did anyone manage to use Serverless as DBT-CLI?

Thanks

[Kaniz_Fatma]

Hi @dbph,

1. • You can disable SSL certificate verification in Databricks CLI by specifying insecure = True in your Databricks configuration file (databrickscfg). This might help bypass the SSL error¹.
   • The Databricks serverless documentation suggests configuring the client to trust the CA file located in /etc/ssl/certs/ca-certificates.crt.

To achieve this, you can run the following command at the beginning of your serverless notebook or job:

import os
os.environ['SSL_CERT_FILE'] = '/etc/ssl/certs/ca-certificates.crt'

[dbph]

Hi, 

thanks for your help!

Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.

I already tried adding the code you posted as a task before the DBT-task. That didn't help. I currently see no way of making the python code run directly in the DBT-task.

Any other ideas? 🙂