Compute Policy Does Not Install Libraries

[Kayla]

Has anyone else run into the issue where applying libraries through a compute policy just completely does not work?
I'm trying to insane some pretty basic Python libraries from PyPI (pytest and paramiko, for example) and it is failing on 13.3 and 14.3 LTS versions.

I am using "Shared" access mode and Unity Catalog enabled.

[Kaniz_Fatma]

Hi @Kayla, In Databricks, compute policies control various aspects of cluster behavior.

When you add libraries to a policy:

• Users can’t install or uninstall compute-scoped libraries on compute that uses this policy.
• Libraries configured through the UI, REST API, or CLI on existing compute are removed the next time the compute restarts.
• Dependency libraries for tasks that use this policy in job compute resources are disabled.
• Please verify that the Python libraries you’re trying to install (such as pytest and paramiko) are compatible with the Databricks runtime versions 13.3 and 14.3 LTS. Sometimes library versions can cause conflicts, so ensure they align with the Databricks runtime you’re using.

[Kayla · Accepted solution]

I was able to install these libraries using a %pip command, so they are definitely compatible with the cluster.
I'm also using an all-purpose cluster doing interactive development work, so the dependent libraries for workflows aren't relevant here.

Interesting change I've seen- the libraries are now installing properly. At a guess, there was some sort of delay after I apply to the policy to the cluster (the cluster was previously on a different policy)