topic Connect Databricks to Airflow in Data Engineering

Connect Databricks to Airflow

T_I — Tue, 10 Dec 2024 19:26:01 GMT

Hi,

I have Databricks on top of aws. I have a Databricks connection on Airflow (mwaa). I am able to conect and execute a Datbricks job via Airflow using a personal access token. I believe the best practice is to conect using a service principal. I understand that I should use the connection id and the secret in order to conect but I get error 401 which I believe it is a result of incorrect Oauth M2M.

Can someone share a light on how it should be done?

Thanks.

Re: Connect Databricks to Airflow

Alberto_Umana — Tue, 10 Dec 2024 19:43:37 GMT

Hi @T_I ,

Have you followed this guide? https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html

You will have to use these 2 parameters from the service principal:

DATABRICKS_CLIENT_ID: The service principal’s client ID.
DATABRICKS_CLIENT_SECRET: The service principal’s secret.

Re: Connect Databricks to Airflow

T_I — Wed, 11 Dec 2024 08:41:44 GMT

I have followed this guide but I do not understand how to implement step 4 - OAuth M2M authentication...
If you could assist, it would be greatly appreciated.

Re: Connect Databricks to Airflow

Alberto_Umana — Wed, 11 Dec 2024 13:29:03 GMT

Hi @T_I,

Instead of the PAT token you have to specify the below settings to be able to use the Service Principal:

For workspace-level operations, set the following environment variables:

DATABRICKS_HOST, set to the Databricks workspace URL, for example https://dbc-a1b2345c-d6e7.cloud.databricks.com.
DATABRICKS_CLIENT_ID
DATABRICKS_CLIENT_SECRET

Re: Connect Databricks to Airflow

T_I — Wed, 11 Dec 2024 13:36:01 GMT

Where should I specify those settings?

Re: Connect Databricks to Airflow

Sloka — Tue, 25 Mar 2025 08:50:32 GMT

https://airflow.apache.org/docs/apache-airflow-providers-databricks/6.9.0/connections/databricks.html