Issue enabling mosaic

[reggie]

Hi,

I am trying to install mosaic on my cluster, but get the error once I use 'enable_mosaic': 

ImportError: cannot import name '_to_java_column' from 'pyspark.sql.functions' (/databricks/spark/python/pyspark/sql/functions/__init__.py)
File <command-1412689069782833>, line 2
      1 from pyspark.sql.functions import *
----> 2 from mosaic import enable_mosaic
      3 enable_mosaic(spark, dbutils)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/__init__.py:1
----> 1 from .api import *
      2 from .core import MosaicLibraryHandler, MosaicContext
      3 from .utils.display_handler import displayMosaic
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/__init__.py:1
----> 1 from .accessors import *
      2 from .aggregators import *
      3 from .constructors import *
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages/mosaic/api/accessors.py:2
      1 from pyspark.sql import Column
----> 2 from pyspark.sql.functions import _to_java_column as pyspark_to_java_column
      4 from mosaic.config import config
      5 from mosaic.utils.types import ColumnOrName

The commands I used to import mosaic are:

%pip install databricks-mosaic

%python
from pyspark.sql.functions import *
from mosaic import enable_mosaic
enable_mosaic(spark, dbutils)

 I have already verified that I have the dependencies required using 

%python pip show databricks-mosaic

which outputs the following:

Name: databricks-mosaic
Version: 0.4.2
Summary: Mosaic: geospatial analytics in python, on Spark
Home-page: https://github.com/databrickslabs/mosaic
Author: Databricks
Author-email: 
License: Databricks License
Location: /local_disk0/.ephemeral_nfs/envs/pythonEnv-d2578267-976a-48ca-ae64-6686226a93e0/lib/python3.10/site-packages
Requires: geopandas, h3, ipython, keplergl, pyspark
Required-by: 

My cluster configuration is as follows, with Photon enabled:

• 2-5 Workers 256-640 GB
• Memory 64-160 Cores
• 1 Driver128 GB Memory, 32 Cores
• Runtime14.3.x-scala2.12
• Unity Catalog
• Photon
• Standard_D32a_v4
• 36–72 DBU/h

I've also tried to restart my cluster to no avail.

Is there a way I can get mosaic successfully enabled to access the spatial functions?

[szymon_dybczak]

Hi @reggie ,

You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet with all reqeuirements:

Installation guide — Mosaic (databrickslabs.github.io)

[szymon_dybczak · Accepted solution]

Hi @reggie ,

You're using wrong runtime. Mosaic 0.4.x series only supports DBR 13.x DBRs. Look at below documentation and make sure that you meet all requirements:

Installation guide — Mosaic (databrickslabs.github.io)

[reggie]

Thank you, I thought mosaic requires a DBR at least 13.x since I didn't get a version error, but this fixed my problem 🙂