topic Re: How do I specify a version of a library different from the Databricks runtime? in Data Engineering

How do I specify a version of a library different from the Databricks runtime?

User16137833804 — Wed, 23 Jun 2021 20:26:15 GMT

Re: How do I specify a version of a library different from the Databricks runtime?

brickster_2018 — Wed, 23 Jun 2021 21:20:04 GMT

The question is not clear. What kind of library are you referring to here?

For Python libraries, it's possible to remove the existing libraries and install the new ones using an init script
For Jar libraries, adding a different version of jar could be very risky.

Re: How do I specify a version of a library different from the Databricks runtime?

sean_owen — Wed, 01 Sep 2021 19:19:25 GMT

For Python libs, the simpler thing is to use %pip install to install them in a notebook. Yes, it could potentially break compatibility with other installed versions. pip will try to figure that out. That could be simpler to deal with than in an init script, and digging for errors on cluster creation.

Re: How do I specify a version of a library different from the Databricks runtime?

Sebastian — Wed, 08 Sep 2021 17:47:11 GMT

the best solution is to store the .whl locally and do a pip install of the local whl while server boots up. this will freeze the library version. if you install from the pip it might impact your production work.