topic Library installation in cluster taking a long time in Data Engineering

Library installation in cluster taking a long time

AyushModi038 — Fri, 17 Feb 2023 14:26:26 GMT

I am trying to install "pycaret" libraray in cluster using whl file.

But it is creating conflict in the dependency sometimes (not always, sometimes it works too.)

My questions are -

1 - How to install libraries in cluster only single time (Maybe from cache). Because it downloads and install them everytime I start the cluster.

It takes around 20 minutes to install this.

2 - How to solve the dependency error and why it is not replicated always?

This might be due to change in numpy version because default runtime has 1.21.5 and after the library installation it changes to 1.19.5 (sometimes).

And the error That i get is

"ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject"

Another issue that follows when above gets resolved (suprisingly) is

'ImportError: Numba needs NumPy 1.20 or less' which also get reolved after I re-run the cell.

Can someone please help??

Re: Library installation in cluster taking a long time

jose_gonzalez — Wed, 22 Feb 2023 22:02:21 GMT

Hi,

Which DBR version are you using? are you installing the library using an init script or once the cluster is up and running, you install it? do you see any error message while trying to install the library? check the driver logs.

Re: Library installation in cluster taking a long time

AyushModi038 — Mon, 27 Feb 2023 09:53:07 GMT

Hi Jose,

Thanks for the help.

Here are th requested details -

DBR Version - 12.1 ML (includes Apache Spark 3.3.1, Scala 2.12)

Installation Mode - Using cluster UI page -> Libarary Tab -> Install New. It installs the libarary everytime the cluster starts.

Error Messages in Driver Logs -

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

It is due to numpy version mismatch

Re: Library installation in cluster taking a long time

Anonymous — Sat, 11 Mar 2023 03:16:41 GMT

Hi @Ayush Modi

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future.

Thank you for your participation and let us know if you need any further assistance!

Re: Library installation in cluster taking a long time

efry — Wed, 04 Oct 2023 16:59:53 GMT

I'm having the exact same problem and its causing issues when I run workflows, too. Please advise, databricks.

Re: Library installation in cluster taking a long time

Spencer_Kent — Tue, 04 Jun 2024 17:27:09 GMT

Can any Databricks pros provide some guidance on this? My clusters that have "cluster-installed" libraries take 30 minutes or more to become usable. I'm only trying to install a handful of CRAN libraries, but having to re-install them every time a cluster starts up is SO painful.

Re: Library installation in cluster taking a long time

shirlyb-melio — Wed, 26 Jun 2024 14:03:17 GMT

I am experiencing a similar issue where a few libraries take 15 minutes to install when running a workflow. Could you please advise if there is a solution for this?

Re: Library installation in cluster taking a long time

Spencer_Kent — Fri, 05 Jul 2024 17:22:18 GMT

@Retired_modWhat about question #1, which is what subsequent comments to this thread have been referring to? To recap the question: is it possible for "cluster-installed" libraries to be cached in such a way that they aren't completely reinstalled every time the cluster is started?

Re: Library installation in cluster taking a long time

fifata — Fri, 14 Mar 2025 18:27:07 GMT

There is a way to bypass this problem.

create a databricks repo and include a folder called "Modules"
download and unpack the whl/tar.gz to a folder (best practice, name the folder as the module itself) of the python module you'd like to install
push that folder inside Modules folder in the databricks repo
in your notebook, where you need to use that module, in the first cell add this -> sys.path.append("<full-path-to-Modules>").
Import your module as usual..

In that way, you wouldn't have to install the wheel every time your cluster starts..