Hello,
I am using asset bundles to build, deploy, create a cluster and install a package in it. The package comes with some extra dependencies that I also want to include in the cluster. Do you know how to specify this extras in the databricks.yml file? Right now I have this:
resources:
jobs:
train:
name: train-ml
tasks:
- task_key: train-ml
existing_cluster_id: ****
spark_python_task:
python_file: "pipelines/train.py"
libraries:
- whl: ./dist/*.whl
