Custom Library's(Unity Catalog Enabled Clusters)

Chris_Shehu
Valued Contributor III

I'm trying to use a custom library that I created from a .whl file in the workspace/shared location. The library attaches to the cluster without any issues and I can it when I list the modules using pip. When I try to call the module I get an error that the module can't be found. I can't seem to find a solution. I tried to manually update the path with the direct locations to the library same issue. I just found an article that says that 3rd party libraries are not usable on unity catalog clusters...The article is a little ambiguous, are all libraries third party? So my custom library won't work? This is a huge draw back if it's true. 

Use Unity Catalog with your Delta Live Tables pipelines - Azure Databricks | Microsoft Learn

  • Init scripts, third-party libraries and JARs are not supported.