topic Re: How do you define PyPi libraries on job level in Asset Bundles? in Data Engineering

How do you define PyPi libraries on job level in Asset Bundles?

jacovangelder — Fri, 28 Jun 2024 11:08:50 GMT

Hello,

Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is there a way to put libraries on job level some other way?

I tried the following:

tasks: - task_key: task1 job_cluster_key: job_cluster notebook_task: notebook_path: ../foo.ipynb - task_key: task2 depends_on: - task_key: task1 job_cluster_key: job_cluster notebook_task: notebook_path: ../foo.ipynb libraries: - pypi: package: pyyaml==6.0.1 - pypi: package: requests==2.31.0 - pypi: package: typing_extensions==4.4.0

Validating the DAB it does not fail, but doesn't work either. It only works when I put the libraries object on the task key level, which feels weird to me. Does that mean we can have different libraries installed for each task? The documentation doesn't really shine any light on this. I can define all libraries on the first task, then I guess the second task will inherit them also, but this feels weird.

I know from DBR 15.x and onwards we can use a requirements.txt workspace, but I am on DBR 14.3 LTS.

I hope someone is able to shine some light on this?

Re: How do you define PyPi libraries on job level in Asset Bundles?

Witold — Mon, 01 Jul 2024 10:54:47 GMT

This is actually the way how job clusters work, you specify dependent libraries on task level.

However, starting with DB CLI v0.222.0 you could try to use complex variables for this kind of configuration.

Re: How do you define PyPi libraries on job level in Asset Bundles?

jacovangelder — Mon, 01 Jul 2024 12:27:55 GMT

Thanks @Witold ! Thought so.

I decided to go with an init script where I install my dependencies rather than installing libraries.

For future reference, this is what it looks like:

job_clusters: - job_cluster_key: job_cluster new_cluster: spark_version: ${var.spark_version} node_type_id: ${var.node_type_id} autoscale: min_workers: ${var.min_workers} max_workers: ${var.max_workers} data_security_mode: SINGLE_USER init_scripts: - workspace: destination: ${workspace.file_path}/resources/init-scripts/init-script.sh

Re: How do you define PyPi libraries on job level in Asset Bundles?

brickster13 — Mon, 30 Mar 2026 08:49:29 GMT

This is a massive headache for my team. We might have interpreted your documentation wrong, so please correct me if that is the case.

Having libraries installed at the task level results in installing the libraries for every task. That is a massive waste of compute, and results in our lakeflow jobs taking way more time than necessary.

On the other hand, it seems that init_scripts is a workaround for this issue. However, in your docs you specify that Databricks does not suggest managing package installs this way, but with little explanation as to why this is not reccommended.

Is there really no option of specifying libraries for a job?

Re: How do you define PyPi libraries on job level in Asset Bundles?

brickster13 — Mon, 30 Mar 2026 08:51:33 GMT

In this community post: https://community.databricks.com/t5/data-engineering/creating-cluster-configuration-with-library-dependency-using/td-p/120838

It seems that job level libraries is allowed?

Personally, both databricks bundle validate and deploy throws no erros if I deploy a bundle with job level libraries, but when I run the job, none of the libraries are actually installed.