Specify extras of library in asset bundles yml file

pablobd — Fri, 15 Dec 2023 10:01:05 GMT

Hello,

I am using asset bundles to build, deploy, create a cluster and install a package in it. The package comes with some extra dependencies that I also want to include in the cluster. Do you know how to specify this extras in the databricks.yml file? Right now I have this:

resources:

jobs:

train:

name: train-ml

tasks:

- task_key: train-ml

existing_cluster_id: ****

spark_python_task:

python_file: "pipelines/train.py"

libraries:

- whl: ./dist/*.whl

Re: Specify extras of library in asset bundles yml file

jessysantos — Mon, 27 May 2024 21:37:30 GMT

Hello @pablobd!

Have you tried declaring these additional dependencies of your wheel file before the wheel file itself? Here's an example snippet for clarification:

resources: jobs: train: name: train-ml tasks: - task_key: train-ml existing_cluster_id: **** spark_python_task: python_file: "pipelines/train.py" libraries: - pypi: package: dependency-1==0.0.1 - pypi: package: dependency-2==0.0.2 - whl: ./dist/*.whl

You can learn more about how to declare Databricks Asset Bundles library dependencies in this following doc: https://docs.databricks.com/en/dev-tools/bundles/library-dependencies.html#databricks-asset-bundles-library-dependencies

Best Regards,
Jéssica Santos

topic Specify extras of library in asset bundles yml file in Machine Learning

Specify extras of library in asset bundles yml file

Re: Specify extras of library in asset bundles yml file