Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 07:03 AM
Thanks Pedro, we did it, for anyone in the future (I added fake host and service principal id's):
1. Modify your databricks.yml so it have the service principal id and the databricks host:
bundle:
name: my_workflow
# Declare to Databricks Assets Bundles that this is a Python project
# This is the interaction with the "pyproject.toml" file
artifacts:
default:
type: whl
build: poetry build
path: .
resources:
jobs:
my_workflow:
name: my_workflow
job_clusters:
- job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
new_cluster:
num_workers: 2
spark_version: "15.3.x-cpu-ml-scala2.12"
node_type_id: Standard_DS3_v2
tasks:
- task_key: my_workflow_pipeline_task
job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
python_wheel_task:
package_name: my_workflow
entry_point: my_workflow_pipeline_task
libraries:
- whl: ./dist/*.whl
permissions:
# If you are using a group, you need to create it in the Databricks workspace
- group_name: "my_group_name"
level: "CAN_MANAGE"
targets:
dev:
mode: development
default: true
workspace:
# Put here the associated workspace url
host: https://adb-0000000000000000.7.azuredatabricks.net
run_as:
# Put here the associated service_principal_name
service_principal_name: 76w4hdge-39a2-0303-45c7-udnr93kvp03f
resources:
jobs:
my_workflow:
job_clusters:
- job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
new_cluster:
num_workers: 2
spark_version: "15.3.x-cpu-ml-scala2.12"
node_type_id: Standard_DS3_v2
permissions:
# If you are using a group, you need to create it in the Databricks workspace
- group_name: "my_group_name"
level: "CAN_MANAGE"
2. Create a .databrickscfg file in the same route where your databricks-cli is installed, so it has the following information:
[my_workflow]
host=https://adb-0000000000000000.7.azuredatabricks.net/
client_id = 76w4hdge-39a2-0303-45c7-udnr93kvp03f
client_secret = tomatoes***************spinach
3. In the terminal just run:
databricks bundle deploy --profile my_workflow
If all was done correctly this should be the output:
(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow
Building default...
Uploading my_workflow-0.1.1-py3-none-any.whl...
Uploading bundle files to /Users/76w4hdge-39a2-0303-45c7-udnr93kvp03f/.bundle/my_workflow/dev/files...
Deploying resources...
Updating deployment state...
Deployment complete!
If the .databrickscfg was not created this could appear:
(.venv) oishiiramen@3301 my_directory % databricks bundle deploy --profile my_workflow
Error: cannot resolve bundle auth configuration: cannot parse config file: open /Users/oishiiramen/.databrickscfg: no such file or directory