How do I specify a version of a library different from the Databricks runtime?

[brickster_2018]

The question is not clear. What kind of library are you referring to here?

• For Python libraries, it's possible to remove the existing libraries and install the new ones using an init script
• For Jar libraries, adding a different version of jar could be very risky.

[sean_owen]

For Python libs, the simpler thing is to use %pip install to install them in a notebook. Yes, it could potentially break compatibility with other installed versions. pip will try to figure that out. That could be simpler to deal with than in an init script, and digging for errors on cluster creation.

[Sebastian]

the best solution is to store the .whl locally and do a pip install of the local whl while server boots up. this will freeze the library version. if you install from the pip it might impact your production work.