topic Re: Custom Library's(Unity Catalog Enabled Clusters) in Data Engineering

Custom Library's(Unity Catalog Enabled Clusters)

Chris_Shehu — Fri, 25 Aug 2023 17:13:43 GMT

I'm trying to use a custom library that I created from a .whl file in the workspace/shared location. The library attaches to the cluster without any issues and I can it when I list the modules using pip. When I try to call the module I get an error that the module can't be found. I can't seem to find a solution. I tried to manually update the path with the direct locations to the library same issue. I just found an article that says that 3rd party libraries are not usable on unity catalog clusters...The article is a little ambiguous, are all libraries third party? So my custom library won't work? This is a huge draw back if it's true.

Use Unity Catalog with your Delta Live Tables pipelines - Azure Databricks | Microsoft Learn

Init scripts, third-party libraries and JARs are not supported.

Re: Custom Library's(Unity Catalog Enabled Clusters)

Chris_Shehu — Sun, 27 Aug 2023 19:04:34 GMT

The documentation referenced above is a little confusing but I've been able to get a library to work successfully. The issue I had was with how structured my library before packaging it into a .whl file.

Re: Custom Library's(Unity Catalog Enabled Clusters)

Szpila — Mon, 28 Aug 2023 08:19:54 GMT

Hello Guys,

I am working on the project where we need to use spark-excel library (Maven) in order to ingest data from excel files. As those 3rd party library are not allowed on shared cluster, do you have any workaround other then using pandas for example.