PabloCSD
Valued Contributor II

Thanks Pedro, we did it, for anyone in the future (I added fake host and service principal id's):

1. Modify your databricks.yml so it have the service principal id and the databricks host:

 

bundle:
  name: my_workflow

# Declare to Databricks Assets Bundles that this is a Python project
# This is the interaction with the "pyproject.toml" file
artifacts:
  default:
    type: whl
    build: poetry build
    path: .

resources:
  jobs:
    my_workflow:
      name: my_workflow
      job_clusters:
        - job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
          new_cluster:
                num_workers: 2
                spark_version: "15.3.x-cpu-ml-scala2.12"  
                node_type_id: Standard_DS3_v2      
      tasks:
        - task_key: my_workflow_pipeline_task
          job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
          python_wheel_task:
             package_name: my_workflow
             entry_point: my_workflow_pipeline_task
          libraries:
            - whl: ./dist/*.whl
      permissions:
        # If you are using a group, you need to create it in the Databricks workspace
        - group_name: "my_group_name"
          level: "CAN_MANAGE"

targets:
  dev:
    mode: development
    default: true
    workspace: 
      # Put here the associated workspace url
      host: https://adb-0000000000000000.7.azuredatabricks.net
    run_as:
      # Put here the associated service_principal_name
      service_principal_name: 76w4hdge-39a2-0303-45c7-udnr93kvp03f
    resources:
      jobs:
        my_workflow:
          job_clusters:
            - job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
              new_cluster:
                num_workers: 2
                spark_version: "15.3.x-cpu-ml-scala2.12"  
                node_type_id: Standard_DS3_v2
          permissions:
          # If you are using a group, you need to create it in the Databricks workspace
          - group_name: "my_group_name"
            level: "CAN_MANAGE"

 

2. Create a .databrickscfg file in the same route where your databricks-cli is installed, so it has the following information:

 

[my_workflow]
host=https://adb-0000000000000000.7.azuredatabricks.net/
client_id = 76w4hdge-39a2-0303-45c7-udnr93kvp03f
client_secret = tomatoes***************spinach

 

3. In the terminal just run:

 

databricks bundle deploy --profile my_workflow

 

If all was done correctly this should be the output:

 

(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow
Building default...
Uploading my_workflow-0.1.1-py3-none-any.whl...
Uploading bundle files to /Users/76w4hdge-39a2-0303-45c7-udnr93kvp03f/.bundle/my_workflow/dev/files...
Deploying resources...
Updating deployment state...
Deployment complete!

 

If the .databrickscfg was not created this could appear:

 

(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow
Error: cannot resolve bundle auth configuration: cannot parse config file: open /Users/oishiiramen/.databrickscfg: no such file or directory

 

 

View solution in original post