I try to import **hdbscan** but it throws this following error

/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py in import_patch(name, globals, locals, fromlist, level)
    156             # Import the desired module. If you’re seeing this while debugging a failed import,
    157             # look at preceding stack frames for relevant error information.
--> 158             original_result = python_builtin_import(name, globals, locals, fromlist, level)
    159 
    160             is_root_import = thread_local._nest_level == 1
 
hdbscan/_hdbscan_linkage.pyx in init hdbscan._hdbscan_linkage()
 
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 80 from PyObject

When I update the numpy version to 1.22.0, I am not getting this error, but it fails to import **umap** which tells the numpy version to be <1.20.

To summarize

Databricks runtime version - 10.1 ML (includes Apache Spark 3.2.0, Scala 2.12)

Python Version - 3.8.10

Python packages installed

umap-learn==0.5.1

numpy==1.22.0

hdbscan==0.8.27

(This version throws error while importing umap)

Python packages installed

umap-learn==0.5.1

numpy==1.20.0

hdbscan==0.8.27

(This version throws the above error)