Databricks Community

Ankitkalra40 · 3 weeks ago

I am working with a client to set up their budget policies and compute governance.

They have a team of Engineers and Scientists that work across different versions of various libraries like python, pandas, numpy, scikit, mlflow etc.

Since each user requires diff versions, having one init script defeats the purpose. Also, any library and the version to be installed is supposed to be validated by InfoSec.

I have proposed a rather niche solution to this that utilizes DBX app as the interface and syncs the approved libraries via Lakebase instance.

We are testing this out but wanted to check if there is something easier available within Databricks at a platform level.

Thanks in advance!

ThomazNeto · 3 weeks ago

Hi Ankit,

Before building anything custom, have a look at three things that already exist in the platform. Together they cover most of what you described.

Workspace base environments. A workspace admin defines a YAML with the serverless environment version and a pinned list of Python dependencies (requirements file, wheels in a UC volume, an internal index-url). Databricks pre-builds and caches it, users pick it from the Environment panel, and admins can star one as the workspace default. You're limited to 10 per workspace, so think "approved profiles" (data-eng, ml-sklearn, ml-mlflow) rather than one per person. Base environments also work on classic compute through "manage dependencies using environments". This is probably the closest thing to what you're prototyping, without the app and the Lakebase sync.
https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment
https://docs.databricks.com/aws/en/compute/serverless/dependencies
Compute policies with libraries. For classic compute, a policy can carry up to 500 libraries (PyPI, wheels in volumes, requirements.txt) that get installed automatically, and the docs are explicit about the side effect you want: "Users can't install or uninstall compute-scoped libraries on compute that use this policy." Databricks also recommends policies over init scripts for library installs. Note this blocks compute-scoped libraries; notebook-scoped %pip still works unless you cut off the index (see next point).
https://docs.databricks.com/aws/en/admin/clusters/policies
Control the source, not the list. The docs say private mirrors like Nexus or Artifactory are supported via --index-url in %pip and in base environment YAML, and admins can set a private repo as the default pip source for serverless. If InfoSec curates what's in the mirror, the approval happens once, at the repository, and every install path inherits it. That's cleaner than approving versions inside Databricks.
https://docs.databricks.com/aws/en/libraries/

One more, since you tagged Unity Catalog: on standard access mode, JARs, Maven coordinates and init scripts need to be in the UC artifact allowlist (MANAGE ALLOWLIST privilege). It doesn't cover PyPI, so for Python the mirror is the control point.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/allowlist

Your app might still make sense as the request/approval UI, but I'd have it write base environments and policy libraries rather than manage installs itself.

Hope this helps.

Thomaz A. Rossito Neto
Principal Data Architect & AI Strategy — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto

Ankitkalra40 · 3 weeks ago

Hi Thomaz,
I agree with most of your recommendations which are in line with the best practices as well.
However, the client is a bit strict about their data policies and Serverless is disabled at their org level.

This is the primary reason a lot of above can not be directly implemented or tested for their requirement.

Is there any other alternative that you can help with? Happy to connect separately on this as well

balajij8 · 3 weeks ago

@Ankitkalra40

You can handle it natively using Compute Policies with Enforced Libraries instead of building a custom DBX app to sync libraries. Workspace admins can build per team compute policies that enforce specific library versions. The key is that users on these policies are completely locked out of installing or uninstalling libraries on this compute. More details here

You set it up directly in the policy's Libraries tab that accepts up to 500 libraries per policy. Instead of using a monolithic init script, you can create dedicated policies per team / dependency matrix eg., one policy mapped to Engineers and another for Scientists

You can pair these policies with Unity Catalog Volumes to address the InfoSec approval process. You can store the InfoSec-validated .whl, .jar and requirements.txt files in a UC Volume and use Unity Catalog grants to restrict READ access to the appropriate teams. More details here

Compute policies can point directly to these volume paths instead of public repositories. It creates a tightly gated, native workflow - InfoSec reviews a library and drops the package into the secure volume, the team's compute policy picks it up and the engineers cannot deviate.

ivanvyd · 3 weeks ago

@Ankitkalra40 since Serverless is disabled, there is one newer native option worth checking: as of the September 17 release, base environments are supported on classic compute in Beta.

I haven't tried it yet, honestly, but as you might want to consider it as an alternative to other answers.

On DBR 19+ with Standard access mode, you can use Dependency mode = Environments and provide admin-managed workspace base environments.

One caveat for the InfoSec requirement: this is not a hard allowlist by itself, because users can still add notebook-level dependencies. If only approved packages must be installable, I'd keep the trust boundary at the package source: enforced compute-policy libraries / approved wheels in UC Volumes, plus a curated private package repository and egress restrictions.

Also worth noting that Environments mode currently targets interactive all-purpose notebooks; classic jobs don't use it yet.

Ivan Vydrin
Lead Software & AI Engineer · Tech Fabric LLC

anuj_lathi · Friday

Skip the custom app and Lakebase sync. On classic compute, compute policies with pinned libraries, scoped per team or persona, cover this at the platform level, and they don't depend on serverless.

Why not a single init script

Databricks recommends compute policies over init scripts for library management, and says not to install libraries via init scripts at all. Widespread init-script use also slows Databricks Runtime upgrades [1]. Policies are the native governance layer.

Recommended pattern

1. One policy per persona or team, with the runtime pinned

Much of the numpy/pandas/scikit-learn/mlflow version spread comes from the Databricks Runtime itself. Pin a fixed runtime version in each policy, using the ML runtime for data science teams [2]. Then grant CAN USE on only the policies a user needs, and remove the unrestricted cluster creation entitlement. Without that entitlement, users can only create compute from their granted policies [2].

2. Put the approved libraries in the policy

Policies can carry libraries that are installed automatically on compute created from them [1]. Pin exact versions (pandas==2.1.4, not pandas). Anything outside the policy then isn't part of a standard cluster build.

3. Make InfoSec approval a pipeline, not an app

Approved packages come from a curated internal mirror (Artifactory/Nexus or similar) or from approved wheels stored in a Unity Catalog volume.
Only a CI service principal can write to that location [3].
Policy changes go through a Git PR, which InfoSec reviews, and are applied with Terraform or the Cluster Policies API. That gives you approval history, versioned policies, and rollback.
Look at the workspace-level default package repository setting to point pip at the internal mirror [4]. Check that it applies to the compute types you use.
To stop users bypassing this with %pip install from public PyPI, block direct egress to public indexes at the network layer and allow only the mirror.

4. Use the standard-access-mode allowlist

On standard access mode compute, libraries and init scripts must be on the metastore allowlist before they can be used [1]. This is a strong control for a strict InfoSec stance. Use standard access mode by default, as Databricks recommends, and fall back to dedicated access mode only when a workload needs it [1].

5. If you still need an init script

Keep it to one small, cluster-scoped script, not a global one, stored in a UC volume (DBR 13.3 LTS and above) [1]. Pin any OS packages you install with it [1]. Then let policies carry the per-team differences: policies can set environment variables and Spark configs [1], so the same script can behave differently per policy.

6. Cost attribution on classic compute

Serverless budget policies don't apply when serverless is off. Instead, enforce fixed custom_tags (team, cost center, project) in each compute policy so usage can be attributed in billing data. Also cap size with limits on node types, autoscaling max, and auto-termination.

Suggested structure

Policy	Runtime	Libraries	Tags
`ds-ml-team-a`	Pinned ML LTS	Approved pinned set A	`team=a`
`de-etl-team-b`	Pinned LTS	Approved pinned set B	`team=b`
`sandbox`	Pinned	Minimal base only, tight size limits	`team=sandbox`

If someone needs a new version, they raise a request. InfoSec approves it, the mirror or volume is updated, and the policy PR merges. Teams then restart their clusters to pick it up.

Docs

The links I've referenced are AWS docs. If the client is on Azure or GCP, the same concepts apply, but use that cloud's doc site and its storage and network service names.