topic Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle in Administration & Architecture

Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle

PabloCSD — Tue, 24 Sep 2024 21:11:37 GMT

How can I connect using a Service Principal Token, I did this, but it is not a PAT:

databricks configure Databricks host: https:// ... Personal access token: ****

I also tried this, but didn't work either:

[profile] host = <workspace-url> client_id = <service-principal-client-id> client_secret = <service-principal-secret>

I tried this way, but nothing (just in case):

databricks configure --aad-token

How can I configure my Databricks workspace so I can deploy DAB in that workspace, but using the service-principal token (for no relying in PAT's)

Best regards, and thank you

#DAB #DatabricksAssetsBundle #ServicePrincipal

Re: Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle

dataeng42io — Wed, 25 Sep 2024 04:29:41 GMT

Hi @PabloCSD,

Long story short you can watch this video where I go step by step on how to set up service principal in azure, grant permissions to workspace and generate a token to itself by doing a machine to machine authentication in the Databricks CLI.

The steps that you need to take to deploy your bundle using service principle.

1.Add service principal to your Databricks Account

2.Give that service principal administration rights to the workspace you want to deploy the DAB

3. Generate a PAT (personal access token) to the service principal. Which you can do in 2 ways.

3.a Either via a Machine to Machine authentication where the service principal generate a PAT to itself. I demonstrate this in the video

3,b or you can generate a token to the sp by using the "on behalf" option providing the principal generating the token has at least the workspace administration writes. On this post there is a solution for this option.

To deploy your bundle using the cli you will use the command

databricks bundle deploy -t <target-name> -p <sp-profile>

The service principle profile needs to have your service principal configured on your ~/.databrickscfg file either with a machine to machine (oauth token) or pat (personal access token)

Hope this helps.

Let me know if you can solve your issue. If any other questions I am here to help

Regards

Pedro

Re: Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle

dataeng42io — Wed, 25 Sep 2024 05:45:59 GMT

Just adding the documentation about authentication -> https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#m2m-auth

Re: Use a Service Principal Token instead of Personal Access Token for Databricks Asset Bundle

PabloCSD — Thu, 26 Sep 2024 14:03:05 GMT

Thanks Pedro, we did it, for anyone in the future (I added fake host and service principal id's):

1. Modify your databricks.yml so it have the service principal id and the databricks host:

bundle: name: my_workflow # Declare to Databricks Assets Bundles that this is a Python project # This is the interaction with the "pyproject.toml" file artifacts: default: type: whl build: poetry build path: . resources: jobs: my_workflow: name: my_workflow job_clusters: - job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster new_cluster: num_workers: 2 spark_version: "15.3.x-cpu-ml-scala2.12" node_type_id: Standard_DS3_v2 tasks: - task_key: my_workflow_pipeline_task job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster python_wheel_task: package_name: my_workflow entry_point: my_workflow_pipeline_task libraries: - whl: ./dist/*.whl permissions: # If you are using a group, you need to create it in the Databricks workspace - group_name: "my_group_name" level: "CAN_MANAGE" targets: dev: mode: development default: true workspace: # Put here the associated workspace url host: https://adb-0000000000000000.7.azuredatabricks.net run_as: # Put here the associated service_principal_name service_principal_name: 76w4hdge-39a2-0303-45c7-udnr93kvp03f resources: jobs: my_workflow: job_clusters: - job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster new_cluster: num_workers: 2 spark_version: "15.3.x-cpu-ml-scala2.12" node_type_id: Standard_DS3_v2 permissions: # If you are using a group, you need to create it in the Databricks workspace - group_name: "my_group_name" level: "CAN_MANAGE"

2. Create a .databrickscfg file in the same route where your databricks-cli is installed, so it has the following information:

[my_workflow] host=https://adb-0000000000000000.7.azuredatabricks.net/ client_id = 76w4hdge-39a2-0303-45c7-udnr93kvp03f client_secret = tomatoes***************spinach

3. In the terminal just run:

databricks bundle deploy --profile my_workflow

If all was done correctly this should be the output:

(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow Building default... Uploading my_workflow-0.1.1-py3-none-any.whl... Uploading bundle files to /Users/76w4hdge-39a2-0303-45c7-udnr93kvp03f/.bundle/my_workflow/dev/files... Deploying resources... Updating deployment state... Deployment complete!

If the .databrickscfg was not created this could appear:

(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow Error: cannot resolve bundle auth configuration: cannot parse config file: open /Users/oishiiramen/.databrickscfg: no such file or directory