topic Re: How can I use custom python library in Azure Databricks? in Data Engineering

How can I use custom python library in Azure Databricks?

Jon — Wed, 20 Oct 2021 08:20:41 GMT

I am trying to access functions in my coreapi.py by importing in the main notebook, but I have error

ModuleNotFoundError: No module named 'coreapi'. I tried by uploading the file into the same folder and I tried creating a python egg and uploading it. How can i import my python library?

Re: How can I use custom python library in Azure Databricks?

Prabakar — Wed, 20 Oct 2021 08:28:00 GMT

Hi @Jon, you need to install the library on the cluster. Please refer to the below document.

https://docs.databricks.com/libraries/cluster-libraries.html#install-a-library-on-a-cluster

Re: How can I use custom python library in Azure Databricks?

Hubert-Dudek — Wed, 20 Oct 2021 09:35:12 GMT

When you go to cluster to edit or you edit individual task you can add own library:

Re: How can I use custom python library in Azure Databricks?

-werners- — Wed, 20 Oct 2021 10:01:16 GMT

There is also the possibility to use Repos file functionality:

https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html