topic NumPy update 1.18-1.21 in Data Engineering

NumPy update 1.18-1.21

KC_1205 — Fri, 04 Mar 2022 13:15:29 GMT

Hi all,

I am planning to update the DB to 9.1 LTS from 7.3 LTS, corresponding NumPy version will be 1.19 and later would like to update 1.21 in the notebooks. At cluster I have Spark version related to the 9.1 LTS which will support 1.19 and notebook will be 1.21. One should expect any compatibility or any other unknown issues? any suggestions will be appreciated.

Thank you

Kiran C

Re: NumPy update 1.18-1.21

-werners- — Sun, 06 Mar 2022 08:07:08 GMT

@Kiran Chalasani , incompatibilities are possible with any upgrade.

Depending on what functionalities of NumPy you use, you will have issues or not.

Some functionalities get deprecated, and others are added. To get a clear view, I suggest looking at the numpy website and see what would change for you.

So check 1.19, 1.20 and 1.21

Re: NumPy update 1.18-1.21

jose_gonzalez — Mon, 25 Apr 2022 19:26:34 GMT

Hi @Kiran Chalasani ,

According to the docs DBR 7.3 LTS comes with Numpy 1.18.1 https://docs.databricks.com/release-notes/runtime/7.3.html and DBR 9.1 LTS comes with Numpy 1.19.2 https://docs.databricks.com/release-notes/runtime/9.1.html

If you need to install a newer version, then you can create a init script or install the version you need. For more details on how to do it, please check the examples in the docs https://docs.databricks.com/libraries/index.html