topic Not able to set run_as service_principal_name in Data Engineering

Not able to set run_as service_principal_name

drag7ter — Wed, 24 Apr 2024 09:19:29 GMT

I'm trying to run:

databricks bundle deploy -t prod --profile PROD_Service_Principal

My bundle looks:

bundle: name: myproject include: - resources/jobs/bundles/*.yml targets: # The 'dev' target, for development purposes. This target is the default. dev: # We use 'mode: development' to indicate this is a personal development copy: # - Deployed resources get prefixed with '[dev my_user_name]' # - Any job schedules and triggers are paused by default # - The 'development' mode is used for Delta Live Tables pipelines mode: development default: true workspace: host: https://myhost.cloud.databricks.com # The 'prod' target, used for production deployment. prod: resources: jobs: my_job1: schedule: pause_status: UNPAUSED # We use 'mode: production' to indicate this is a production deployment. # Doing so enables strict verification of the settings below. mode: production workspace: host: https://prod-host.cloud.databricks.com root_path: /Users/application_id_of_my_service_principal/.bundle/${bundle.name}/${bundle.target} run_as: service_principal_name: PROD_Service_Principal

And I'm getting this error:

Updating deployment state... Error: terraform apply: exit status 1 Error: cannot create job: 'PROD_Service_Principal' cannot be set as run_as_username, either because the user does not exist or is not a service principal.

Why I'm getting this error?

Why it tells me "can not set as run_as_username" I'm specifying not user_name but service_principal_name?

This service principal name exists and active in all workspaces

Re: Not able to set run_as service_principal_name

drag7ter — Wed, 24 Apr 2024 09:51:38 GMT

I want to add that even if I specify:

run_as: user_name: c250831b-5a2a-4461-a855-83b9102f797e

It doesn't work:

Error: cannot create job: 'c250831b-5a2a-4461-a855-83b9102f797e' cannot be set as run_as, either because the user does not exist or it is not active.

Re: Not able to set run_as service_principal_name

drag7ter — Wed, 24 Apr 2024 12:55:27 GMT

In my case I replaced alias PROD_Service_Principal with id c250831b-5a2a-4461-a855-83b9102f797e and it works. Not intuitive, probably this is a bug in CLI ot bundles

service_principal_name: c250831b-5a2a-4461-a855-83b9102f797e

Re: Not able to set run_as service_principal_name

reidwil — Tue, 03 Sep 2024 00:03:21 GMT

Building on this situation, I am seeing if I deploy a job using a service principal this way, I am getting something prepended to the job like `[dev f46583c2_8c9e_499f_8d41_823332bfd4473] `. Is there a different way for me via bundling to change this?

Re: Not able to set run_as service_principal_name

thibault — Tue, 03 Sep 2024 13:16:18 GMT

This is a normal preset for the development mode. You can overwrite it : Databricks Asset Bundle deployment modes - Azure Databricks | Microsoft Learn

Or change the mode to production.