How do you define PyPi libraries on job level in Asset Bundles?

[jacovangelder]

Hello,

Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is there a way to put libraries on job level some other way?

I tried the following:

tasks:
  - task_key: task1
    job_cluster_key: job_cluster
    notebook_task:
      notebook_path: ../foo.ipynb
  - task_key: task2
    depends_on:
        - task_key: task1
    job_cluster_key: job_cluster
    notebook_task:
      notebook_path: ../foo.ipynb
libraries:
- pypi:
    package: pyyaml==6.0.1
- pypi:
    package: requests==2.31.0
- pypi:
    package: typing_extensions==4.4.0

Validating the DAB it does not fail, but doesn't work either. It only works when I put the libraries object on the task key level, which feels weird to me. Does that mean we can have different libraries installed for each task? The documentation doesn't really shine any light on this. I can define all libraries on the first task, then I guess the second task will inherit them also, but this feels weird. 

I know from DBR 15.x and onwards we can use a requirements.txt workspace, but I am on DBR 14.3 LTS.

I hope someone is able to shine some light on this? 

[Witold]

This is actually the way how job clusters work, you specify dependent libraries on task level.

However, starting with DB CLI v0.222.0 you could try to use complex variables for this kind of configuration.

[jacovangelder]

Thanks @Witold ! Thought so. 

I decided to go with an init script where I install my dependencies rather than installing libraries. 

For future reference, this is what it looks like:

job_clusters:
  - job_cluster_key: job_cluster
    new_cluster:
      spark_version: ${var.spark_version}
      node_type_id: ${var.node_type_id}
      autoscale:
          min_workers: ${var.min_workers}
          max_workers: ${var.max_workers}
      data_security_mode: SINGLE_USER
      init_scripts:
        - workspace:
            destination: ${workspace.file_path}/resources/init-scripts/init-script.sh