Specify extras of library in asset bundles yml file

[pablobd]

Hello,

I am using asset bundles to build, deploy, create a cluster and install a package in it. The package comes with some extra dependencies that I also want to include in the cluster. Do you know how to specify this extras in the databricks.yml file? Right now I have this:

resources:

  jobs:

  train:

    name: train-ml

    tasks:

      - task_key: train-ml

        existing_cluster_id: ****

        spark_python_task:

        python_file: "pipelines/train.py"

    libraries:

        - whl: ./dist/*.whl

[jessysantos]

Hello @pablobd!

Have you tried declaring these additional dependencies of your wheel file before the wheel file itself? Here's an example snippet for clarification:

resources:
  jobs:
  train:
    name: train-ml
    tasks:
      - task_key: train-ml
        existing_cluster_id: ****
        spark_python_task:
        python_file: "pipelines/train.py"
    libraries:
        - pypi:
          package: dependency-1==0.0.1
        - pypi:
          package: dependency-2==0.0.2
        - whl: ./dist/*.whl

You can learn more about how to declare Databricks Asset Bundles library dependencies in this following doc: https://docs.databricks.com/en/dev-tools/bundles/library-dependencies.html#databricks-asset-bundles-...

Best Regards,
Jéssica Santos