Unity Catalog shared cluster & third-party1 libraries

[RicksDB]

Hello,

I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries.

 

Nevertheless, I tried doing imports and using pip install and both were working correctly. What is considered a third-party1 library? Will pip install continue to be supported in the future?

Thanks!

[Aviral-Bhardwaj]

yes this is something new in the databricks

here you have to add the library in the account level instead of the cluster level by that you can whitelist those packages

you can use custom or no isolation cluster , it will work there

[MattG1]

Hi Aviral-Bhardwaj-ji, I have seen a couple of mentions about Whitelisting of libraries on the Account but have struggled to find any detail about how to actually do this.

Like RickDB (OP) I have been exploring the idea of 'whitelisting' libraries on the Shared Isolation Cluster but apart from the odd mention of it I have failed find useful documentation about it.

To clarify my understanding of what is meant by "Account"; I am making an assumption that you are referring to the Account accessed from the Admin console (i.e. the location where the metastore is defined)? Are we talking about the same thing? If so, I can't see any means within the UI to whitelist libraries.

Also just to say, I had wondering if the restrictions on 3rd party libraries could be circumvented by using the Global Inits - but through some tinkering realised Global Inits don't work on Shared Clusters(but do on Single Clusters)

Are there any useful resources you can reference?

Thanks

[Aviral-Bhardwaj]

Why don't you send me request here - https://www.linkedin.com/in/aviralb/

and we will connect and discuss your whole thing here, it will be a great talk, and we can share our experiences and queries on databricks

@Matthew Gascoyne​  I will also learn something from you and you will also learn