topic How to add shared libs in Data Engineering

How to add shared libs

MikeGo — Sat, 18 Jan 2025 22:57:21 GMT

Hi team,

I want to add some shared libs which might be used by many repos, e.g. some util functions which might be used by any repos.

1. What is the recommended way to add those libs? E.g. create a separate repo and reference it in another repo?

2. How to control the permission?

3. How to control diff versions?

4. How to share from diff workspaces?

Thanks

Alberto_Umana — Sun, 19 Jan 2025 13:43:43 GMT

You can use cluster libraries so whenever interacting with repos libraries are available. Also the cluster can be shared to others. https://docs.databricks.com/en/libraries/cluster-libraries.html.

About permissions you can control them over cluster permissions too.

MikeGo — Tue, 21 Jan 2025 17:48:05 GMT

Hi @Alberto_Umana , thanks for response. What if each job is using a job cluster?

radothede — Wed, 22 Jan 2025 10:28:53 GMT

Typically, You specify shared libraries in init script. From there, init script will be executed for each job compute, ensuring lib consistency.

The other way - You could use a job cluster policy and specify desired libraries that will be provided during the cluster setup.

Best,

Rado.