Is it Possible to Install Python Libraries or any Python Wheels on the New Databricks Free Account

[cpatte7372]

Hi Community,

I have been informed that it's not possible to install either Python libraries, Python wheels or MAVEN libraries on the new Databricks Free account - It is possible with the old Databrick Community Edition

Can someone please confirm that is the case. If that is the case, can someone let me know what are the alternatives?

Thanks

Carlton

[szymon_dybczak]

Hi @cpatte7372 ,

Serverless compute has some limiations. For instnace, you can't install cluster scoped libraries etc.Take a look at full list here:

- Compute policies
- Compute-scoped init scripts
- Compute-scoped libraries, including custom data sources and Spark extensions. Use notebook-scoped libraries instead.
- Instance pools
- Compute event logs
- Most Apache Spark compute configurations. For a list of supported configurations, see Configure Spark properties for serverless notebooks and jobs.
- Environment variables. Instead, Databricks recommends using widgets to create job and task paramers

Serverless compute limitations - Azure Databricks | Microsoft Learn

But you can install notebook-scoped libraries, like I did below:

[cpatte7372]

 Hi szymon_dybczak

Thanks for getting in touch.

So, if I read you correctly I can do a PIP install, for example 'pip install kafka-python' however, if I created my own python wheel would I not be able to install it, correct?

[szymon_dybczak · Accepted solution]

Hi @cpatte7372 ,

You can still install python wheel files. For instance, I created following simple wheel file:

Then I uploaded it to my Unity Catalog volume and I was able to install it using pip:

%sh
pip install /Volumes/workspace/default/my_volume/hello_pkg-0.1-py3-none-any.whl

[cpatte7372]

szymon_dybczak

This is fantastic.

Thank you so much.

[szymon_dybczak]

No problem @cpatte7372 , glad that I could help 🙂