Specify extras of library in asset bundles yml file

pablobd
Contributor II

‎12-15-2023 02:01 AM

Hello,

I am using asset bundles to build, deploy, create a cluster and install a package in it. The package comes with some extra dependencies that I also want to include in the cluster. Do you know how to specify this extras in the databricks.yml file? Right now I have this:

resources:
  jobs:
  train:
    name: train-ml
    tasks:
      - task_key: train-ml
        existing_cluster_id: ****
        spark_python_task:
        python_file: "pipelines/train.py"
    libraries:
        - whl: ./dist/*.whl
0 Kudos

jessysantos
Databricks Employee
Databricks Employee

‎05-27-2024 02:37 PM

Hello @pablobd!

Have you tried declaring these additional dependencies of your wheel file before the wheel file itself? Here's an example snippet for clarification:

resources:
  jobs:
  train:
    name: train-ml
    tasks:
      - task_key: train-ml
        existing_cluster_id: ****
        spark_python_task:
        python_file: "pipelines/train.py"
    libraries:
        - pypi:
          package: dependency-1==0.0.1
        - pypi:
          package: dependency-2==0.0.2
        - whl: ./dist/*.whl

You can learn more about how to declare Databricks Asset Bundles library dependencies in this following doc: https://docs.databricks.com/en/dev-tools/bundles/library-dependencies.html#databricks-asset-bundles-...

Best Regards,
Jéssica Santos


0 Kudos