Error: cannot create job: 'SERVICE_PRINCIPAL_NAME' cannot be set as run_as_username

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi Team,

I am trying to deploy workflows to Databricks Workspace via Databricks Asset Bundle (DAB) using Azure Service Principle. Below is my databricks.yml file which i am using for DAB.
I am replacing the "SERVICE_PRINCIPAL_NAME" variable in my Jenkins pipeline before running the data bricks cli commands.

bundle:
  name: test
  version: 0.0.1
  terraform:
    exec_path: '/opt/tf_binaries/terraform'

include:
  - resources/*.yml

targets:
  # The 'dev' target, for development purposes. This target is the default.
  dev:
    mode: production
    default: true
    workspace:
      host: https://adb-xxxxx.azuredatabricks.net
      root_path: Workspace/
      artifact_path: /Workspace/artifacts
      file_path: Workspace/
      state_path: /Workspace/state
    run_as:
      service_principal_name: SERVICE_PRINCIPAL_NAME

I am using below commands:

databricks bundle validate -t dev -p DEFAULT & databricks bundle deploy -t dev -p DEFAULT

I am able to validate the bundle successfully but when trying to deploy getting the below error

Error: cannot create job: 'SERVICE_PRINCIPAL_NAME' cannot be set as run_as_username, either because the user does not exist or is not a service principal.

If, i dont deploy the workflow files (.yml files under resources folder) i am able to deploy successfully without any issue.

Kindly suggest on how to deploy the workflow with service_principle.